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Preface 


Battelle  Memorial  Institute  and  James  Gregory  Associates  Inc.  in  support  of  the  Supportability 
Investment  Decision  Analysis  Center  (SID AC)  performed  this  effort,  imder  Subtask  1 1  of  Task 
Order  123  of  Air  Force  Contract  Number  F33657-92-D-2055. 

This  effort  surveyed  the  state-of-the-art  of  commercial-off-the-shelf  business-process  support 
tools  in  late  1996  and  early  1997.  It  then  recommended  strategies  for  tool  development  and 
deployment  to  effect  process  improvement  in  the  Air  Force  S&T  community.  It  represents  only 
a  snapshot  of  a  rapidly  changing  commercial  market.  As  such,  it  makes  no  long-term 
recommendations  for  tool  selection,  but  highlights  tool  features  and  capabilities  whose 
development  warrants  continued  monitoring.  It  also  recommends  areas  of  further  Government 
research,  which  could  augment  commercial  developments  in  meeting  Air  Force  needs. 

To  determine  required  tool  capabilities,  this  effort  included  extensive  interviews  with  program 
managers  at  four  Air  Force  laboratories.  The  interviews,  in  essence,  asked  the  participants  to 
describe  their  needs  for  tools  to  do  business  in  ways  in  which  they  had  they  had  yet  to  use  or 
study.  The  interviews  served  as  much  as  an  education  in  IPPD  for  them  as  it  did  as  an 
information  source  for  this  study.  The  IPPD  initiative  is,  in  part,  an  attempt  to  bring  more 
standardization  and  repeatability  to  the  management  of  Air  Force  research  and  development.  Yet 
research  and  development  is  a  highly  creative  and  subjective  process.  The  general  attitude  of  the 
participants  could  be  described  as  enthusiastic  for  the  opportimity  to  better  document  R&D 
decisions  and  rationale;  and  to  better  communicate  between  team  members,  through  the  chain  of 
command,  and  during  phase  transitions.  At  the  same  time,  their  attitude  was  generally  skeptical 
about  how  well  R&D  decision-making  could  be  reduced  to  business  formulas.  A  tool  is  only  as 
good  as  the  data  it  uses,  and  data  often  comes  from  personal  judgment  when  exploring  the 
unknown.  Other  concerns  of  the  participants  included  the  resources  required  to  procure, 
maintain,  and  train  on  the  tools. 

The  IPPD  initiative  will  effect  such  a  culture  change  in  the  AFMC  S&T  community  that  the  tools  used  to 
implement  it  will  in  large  part  determine  whether  it  is  embraced  positively  or  negatively.  The  long-term 
relevance  of  this  study  is,  perhaps,  the  recording  of  the  participants’  reactions  to  each  of  the  tool  areas  to 
which  they  were  introduced.  This  report  captures  those  reactions  formally  in  lists  of  “major”  and  “core” 
features  desired  for  the  tools,  and  less  formally  but  more  informatively  by  documenting  their  comments 
to  the  “warm-up”  questions. 


Executive  Summary 


The  purpose  of  this  task  was  to  identify,  prototype  and  demonstrate  tools  and  methods  that 
support  the  AFMC/ST  Science  and  Technology  (S&T)  Integrated  Product  and  Process 
Development  (IPPD)  initiative.  This  effort  was  intended  to  provide  critical  research  and 
development  to  establish  the  viability  of  a  tools  development  strategy.  The  intent  is  to  deploy 
Commercial-off-the-Shelf  (COTS)  software  products  that  will  aid  the  S«&;T  community  in 
implementing  the  IPPD  strategy  and,  in  turn,  reduce  program  management  and  operation  costs, 
particularly  in  the  6.3  demonstration  phase.  The  assessment  looked  at  tools  in  four  areas: 

1 .  (Technology)  Requirements  Collection,  Organization,  and  Analysis; 

2.  (Value  Judgment  via)  Group  Consensus; 

3.  (Program  Management)  Workflow;  and 

4.  (Design)  Value  Analysis. 

This  effort  had  three  general  objectives:  (1)  Gather  user  needs  for  tools  to  support  the 
implementation  of  the  IPPD  process.  (2)  Provide  a  market  assessment  to  determine  whether 
commercial  tools  exist  that  could  support  these  needs.  (3)  Recommend  a  tools  development 
strategy  for  deploying  tools  throughout  the  Air  Force  S&T  community. 

The  first  objective  of  this  effort  was  to  gather  user  needs  for  tools  and  methods  to  support  IPPD 
implementation.  The  approach  chosen  to  accomplish  this  objective  was  an  interview 
methodology  using  Ventana  GroupSystems  software.  A  two-day  structured  interview  was 
developed  that  included  an  introduction  to  the  S&T  IPPD  process  and  segments  for  each  of  the 
four  tool  areas  being  researched.  Interviews  were  conducted  with  Wright  Laboratory,  Armstrong 
Laboratory,  Phillips  Laboratory,  and  Rome  Laboratory  personnel.  Program  managers  from 
various  S&T  programs  participated  in  the  interviews  and  provided  a  wealth  of  information  to  the 
research  team. 

Through  the  use  of  GroupSystems,  the  structured  interviews  were  exchanges  of  information 
between  the  research  team  and  the  participating  program  managers.  Specifically,  the  research 
team  first  presented  information  to  the  participants  on  the  S&T  IPPD  process  and  then  on  each  of 
the  tool  areas  being  researched.  After  each  presentation  segment,  the  participants  were  asked  to 
provide  feedback  on  the  information  they  had  just  received.  Throughout  the  entire  interview,  the 
participants  were  also  able  to  enter  their  thoughts  and  comments  about  IPPD  or  relevant  subjects 
into  the  system.  The  GroupSystems  software  captured  all  of  the  input  in  electronic  form. 

The  second  objective  of  this  effort  was  to  determine  if  commercial  tools  exist  that  could  support 
the  user  needs  identified  during  the  interviews.  The  assessment  showed  that  tools  are  available 
which  could  be  tailored  to  support  the  process  in  all  areas  except  value  analysis.  That  tool  area 
requires  a  core,  integrating  toolkit  that  could  interface  to  a  variety  of  design  and  analysis 
applications. 


IV 


This  report  presents  the  results  of  the  interviews  and  the  market  analyses  for  the  four  tool  areas. 

It  became  clear  from  the  data  that  users’  needs  vary  and  that  no  single  tool  in  any  area  is  likely  to 
satisfy  the  needs  of  every  program.  However,  it  was  also  clear  from  the  data  that  users  share 
many  needs,  and  this  commonality  represents  core  capabilities  that  could  be  used  as  a  starting 
point  for  selecting  tools. 

No  recommendations  were  made  for  the  use  of  specific  tools  in  any  area.  Rather,  the  research 
findings  are  presented  in  a  maimer  that  allows  program  managers  to  select  tools  based  on  their 
particular  needs. 

The  third  objective  of  this  effort  was  to  recommend  a  tools  development  strategy  for  deploying 
tools  throughout  the  S&T  community.  As  such,  this  report  is  not  a  software  development  plan, 
but  a  strategy  for  selecting  tools,  customizing  them,  and  integrating  them  over  time  into  specific 
S&T  programs  having  specific  objectives.  The  essence  of  the  strategy  is: 

•  Select  pilot  programs  in  which  to  first  implement  the  S&T  IPPD  process. 

•  Minimize  the  commitment  to  customize  or  combine  tools  until  the  need  is  imminent  in  an 
S&T  program. 

•  Monitor  market  development  regarding  the  key  features  and  capabilities  identified  by 
users  during  the  interviews  of  this  study. 

•  Seed  prototyping  software  developnient  in  value  analysis  because  the  marketplace  is  only 
now  recognizing  this  tool  area  as  a  potential  product  category. 

•  Encourage  and  participate  in  standards  development,  particularly  in  the  areas  of  web- 
enabled  workflow,  requirements  analysis,  and  security. 

•  Form  a  Tools  Working  Group  to  track,  assess,  and  report  on  tool  developments  as  related 
to  the  S&T  IPPD  process. 

The  strategy  delineated  in  this  report  represents  the  first  step  in  deploying  tools  and  methods  to 
assist  program  managers  in  implementing  the  S&T  IPPD  process.  By  employing  IPPD 
principles  and  practices,  the  S&T  culture  can  move  away  from  the  historic  performance-at-any- 
cost  approach  to  technology  development  and  application,  toward  a  new,  more  cost-effective  and 
risk-managed  approach. 
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1.  Introduction 


1.1  Background 

The  Science  and  Technology  (S&T)  Integrated  Product  and  Process  Design  (IPPD)  initiative  was 
launched  in  1993  at  the  direction  of  AFMC/ST.  The  fundamental  goal  of  this  initiative  is  to  use 
IPPD  principles  and  practices  to  better  leverage  S&T  development  efforts.  The  initiative  intends 
to  move  the  S&T  culture  away  from  the  historic  performance-at-any-cost  approach  to  technology 
development  and  application  toward  a  new,  more  cost-effective  and  risk-managed  approach. 

The  IPPD  initiative  is  aimed  at  making  technology  more  affordable.  The  objective  is  to  ensure 
that  only  the  highest  value  technology  products  are  implemented  into  Air  Force  weapon  systems 
and  their  support  infrastructure.  The  initiative,  therefore,  seeks  to  make  IPPD  a  part  of  everyday 
life  in  the  Air  Force  S&T  community  so  that  manufacturing-process  issues  and  life-cycle-support 
issues  are  routinely  balanced  with  product-performance  potential,  much  earlier  in  the  technology- 
development  cycle. 

The  S&T  community  includes  AFMC/ST,  The  Air  Force  Office  of  Scientific  Research 
(AFOSR),  Armstrong  Laboratory  (AL),  Phillips  Laboratory  (PL),  Rome  Laboratory  (RL),  and 
Wright  Laboratory  (WL). 

In  the  past,  IPPD  has  often  been  ignored  or  ineffectively  included  in  S&T  programs  due  to 
factors  such  as  (1)  the  emphasis  on  performance  and  innovation  over  cost;  (2)  the  attitude  that 
implementation  costs  were  the  customers'  problem;  and  (3)  the  lack  of  IPPD  training,  methods, 
and  tools  available  to  S&T  program  managers.  Therefore,  the  S&T  IPPD  initiative  is  focused  on 
overcoming  these  deficiencies.  The  immediate  users  of  the  S&T  IPPD  process  will  be  the  Air 
Force  S&T  community,  and  the  ultimate  beneficiaries  will  be  the  S&T  customers  in  the  Air 
Force  operational  commands  and  support  infrastracture. 


1.2  Purpose 

The  purpose  of  this  requirements-definition-and-tools-deployment  strategy  is  to  provide  an 
overall  strategy  for  identifying  and  deploying  tools  to  support  S&T  IPPD  implementation.  The 
intent  is  to  deploy  Commercial-Off-The-Shelf  (COTS)  software  products  that  will  aid  the  S&T 
community  in  implementing  the  IPPD  strategy,  particularly  in  the  6.3  demonstration  phase,  and, 
in  turn,  reduce  program  management  and  operation  costs.  As  such,  this  report  is  not  a  software 
development  plan,  but  a  strategy  for  selecting  tools,  customizing  them,  and  integrating  them  over 
time  in  the  context  of  specific  S&T  programs  which  are  trying  to  achieve  particular  objectives. 
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The  four  tool  areas  researched  were: 


1.  (Technology)  Requirements  Collection,  Organization,  and  Analysis; 

2.  (Value  Judgment  via)  Group  Consensus; 

3.  (Program  Management)  Workflow;  and 

4.  (Design)  Value  Analysis. 

The  first  objective  of  this  effort  was  to  gather  user  needs  for  tools  and  methods  to  support  6.3 
program  managers  in  the  implementation  of  IPPD.  The  approach  chosen  to  accomplish  this 
objective  was  an  interview  methodology  using  Ventana  GroupSy stems  software.  A  two-day 
structured  interview  was  developed  that  included  an  introduction  to  the  S&T  IPPD  process  and 
segments  for  each  of  the  four  tool  areas  being  researched. 

The  research  team  believed  that  in  order  to  reach  the  first  objective,  the  interviews  would  have  to 
be  information-sharing  sessions.  Clearly,  before  the  program  managers  could  provide  their  needs 
for  tools  and  methods  to  implement  the  S&T  IPPD  process,  they  would  have  to  imderstand  the 
process.  Through  the  use  of  Ventana  GroupSystems,  the  interviews  were  an  exchange  of 
information  between  the  research  team  and  the  participating  program  managers.  Specifically,  the 
research  team  first  presented  information  to  the  participants  on  the  S&T  IPPD  process  and  then 
on  each  of  the  tool  areas  being  researched.  After  each  presentation  segment,  the  participants 
were  asked  to  provide  feedback  on  the  information  they  had  just  received.  Throughout  the  entire 
session,  the  participants  were  also  able  to  enter  into  the  system  their  thoughts  and  comments 
about  IPPD  and  relevant  subjects.  The  GroupSystems  software  captured  all  of  the  input  in 
electronic  form. 

The  research  team  believed  that  this  interactive  approach  to  gathering  needs  for  tools  and 
methods  offered  several  benefits.  First,  the  opportunity  to  expose  program  managers  to  the  S&T 
IPPD  process  would  enhance  the  overall  S&T  IPPD  initiative.  Second,  this  method  of  collecting 
user  needs  would  increase  user  buy-in  of  the  tools  recommended  for  use  and  of  the  S&T  IPPD 
initiative  itself  Third,  the  information  collected  would  help  to  support  the  conclusions  of  the 
research  team  regarding  features  and  capabilities  that  tools  should  have  in  order  to  aid  IPPD 
implementation.  Finally,  the  GroupSystems  software  provides  the  capability  to  capture  all  of  the 
user  interaction  electronically,  thereby  providing  a  complete  transcript  of  the  interviews  —  a 
capability  usually  not  possible  with  traditional  minutes  taken  by  a  scribe. 

Interviews  were  conducted  with  the  organizations  shown  in  table  1.  The  Air  Force  Technical 
Manager  arranged  the  interviews  with  appropriate  S&T  program  managers  and  representatives 
from  laboratory  management. 
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Table  1.  Organizations  Interviewed 
for  IPPD  Tool  Requirements 


Organization 

Interview  Date 

Wright  Laboratory 

5-6  June  1996 

Wright  Laboratory 

19  -  20  June  1996 

Armstrong  Laboratory 

14-15  August  1996 

Phillips  Laboratory 

18-19  September  1996 

Rome  Laboratory 

16-17  October  1996 

The  second  objective  in  this  research  was  to  conduct  a  market  assessment  to  determine  whether 
COTS  software  products  were  available  to  support  the  implementation  of  IPPD.  Once  the  user 
needs  were  collected,  the  research  team  was  able  to  use  the  data  to  determine  evaluation  criteria 
for  these  products.  It  must  be  stressed  that  the  research  team  strongly  believed  it  to  be  unlikely 
that  a  single  tool  from  a  tool  area  would  satisfy  all  research  needs.  In  fact,  the  results  of  this  task 
show  that  each  group  of  users  has  different  needs  depending  on  the  type  of  research  being 
accomplished  and  the  maturity  of  the  technology  being  developed.  Therefore,  no  pretense  was 
made  in  this  report  to  suggest  specific  tools  for  use  by  program  managers.  Rather,  this  effort  has 
identified  candidate  tools  in  each  of  the  four  areas  and  evaluated  them  against  the  needs 
identified  in  the  structured  interviews.  The  results  are  presented  so  that  program  managers  can 
see  which  tools  satisfy  their  particular  needs  at  any  given  time. 


1.3  About  This  Report 

This  document  is  organized  into  five  sections. 

This  section.  Section  1.0,  has  provided  background  and  introductory  information  about  this 
effort. 

This  section  also  described  the  methodology  chosen  to  conduct  this  research  and  the  rationale  of 
the  research  team  for  choosing  it. 

Section  2.0  describes  the  approach  and  protocol  used  to  conduct  the  structured  interviews  at  the 
laboratories.  Included  is  a  discussion  of  the  interview  agenda,  a  summary  of  the  presentations, 
and  a  description  of  the  scenario-based  approach  used  to  illustrate  the  S&T  IPPD  process. 

Section  3.0  presents  the  results  of  the  structured  interviews  and  the  market  assessments  for  each 
of  the  four  tool  areas. 

Section  4.0  makes  recommendations  for  future  tools  and  methods  development. 

Section  5.0  summarizes  the  major  findings  of  this  report. 
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2.  Approach  and  Interview  Protocol 


This  project  set  out  to  ask  program  managers  what  kinds  of  tools  and  methods  they  would  need 
to  implement  a  process  they  had  yet  to  use  or  study.  The  approach,  therefore,  had  to  provide  a 
substantial  amount  of  information  about  the  IPPD  process,  itself,  to  laboratory  program 
managers.  As  a  result,  an  interview  protocol  was  developed  that  included  a  series  of  moderately 
in-depth  presentations.  The  approach  was  to  provide  sufficient  information  about  IPPD  in 
digestible  portions,  one  piece  at  a  time,  so  that  the  attendees  could  focus  on  the  tools  and 
methods  needed  to  implement  each  portion. 

In  order  to  facilitate  the  interviews,  the  research  team  used  Ventana  GroupSystems  software. 
This  groupware  system  consists  of  software  that  runs  simultaneously  across  a  set  of  laptop 
computers.  The  typical  setup  supports  up  to  ten  participants  along  with  a  facilitator  and  a 
technographer.  The  technographer  runs  the  system,  sends  appropriate  information  to  the 
participants’  computers,  and  captures  salient  discussion  information  that  is  not  being  entered  by 
any  of  the  participants.  The  software  provides  for  “comment  cards,”  which  allow  participants  to 
type  in  questions  or  comments  at  any  point  during  the  presentation.  The  comment  cards  are 
structured  topically  and  are  usually  titled  with  either  a  question  or  the  functional  area  on  which 
feedback  is  being  solicited.  As  participants  enter  and  submit  their  comments,  everyone  linked 
with  the  system  can  see  the  comments.  Each  comment  is  numbered  for  tracking,  but  its  author 
remains  anonymous.  Participants  can  respond  to  questions  or  comments  by  referring  to  the 
comment  number  or  by  posting  a  “yellow  sticky”  to  the  left  of  a  comment.  The  technographer 
controls  what  the  participants  can  see  and  do  at  any  time.  The  facilitator  guides  the  participants 
through  the  interview  segments  and  the  use  of  the  software. 

This  section  presents  the  organi2ation  and  content  of  the  interview  protocol.  First,  the  interview 
agenda  is  reviewed.  The  remaining  paragraphs  cover  in  detail  all  of  the  presentations  given 
during  the  two-day  interview  sessions  and  a  discussion  of  the  effectiveness  of  this  approach. 


2.1  Interview  Agenda 

The  research  team  constructed  the  interview  protocol  around  the  premise  that  the  interviews 
would  be  information  exchanges  between  the  team  and  the  participants.  As  noted  above,  the 
participants  had  little  exposure  to  the  S&T  IPPD  process;  thus,  a  moderately  in-depth 
introduction  was  the  first  part  of  the  interview.  As  with  all  the  presentations,  the  GroupSystems 
software  was  available  for  the  participants  to  enter  comments  they  deemed  to  be  appropriate. 
After  the  S&T  IPPD  process  overview,  each  tool  area  was  covered.  The  general  format  of  the 
agenda  for  the  tool  area  presentations  was  as  follows: 

1 .  Present  information  on  the  tool  area  and  its  relationship  to  the  S&T  IPPD  process; 

2.  Present  a  conceptual  demonstration  of  how  a  tool  might  aid  the  S&T  IPPD  process; 
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3 .  Ask  the  participants  a  series  of  “warm-up”  questions  to  stimulate  thinking  about  their 
needs  for  a  tool  in  this  area; 

4.  Solicit  feedback  in  terms  of  needs,  features,  or  capabilities  that  a  tool  in  this  area 
should  have;  and 

5.  Review  and  comment  on  the  resulting  list  of  needs,  features,  and  capabilities. 
Prioritize  these  using  a  voting  process. 


The  research  team  had  planned  to  adjust  the  interview  agenda  as  they  analyzed  participant 
feedback  from  each  session.  This  they  did,  particularly  with  the  warm-up  questions  and  the  tools 
presentations. 

The  warm-up  questions  were  changed  after  the  first  two  interviews  because  some  of  the 
questions  were  too  leading  and  others  simply  failed  to  elicit  the  desired  information.  These 
changes  were  not  seen  as  a  hindrance  to  the  research  objectives  since  the  warm-up  questions 
were  a  precursor  to  the  heart  of  the  effort  --  soliciting  user  needs  for  specific  features  of  the  tools. 

As  a  result  of  participant  feedback,  the  presentations  evolved  to  include  a  conceptual 
demonstration  of  what  tools  might  do  to  aid  S&T  IPPD  implementation.  The  conceptual 
demonstration  was  an  end-to-end  scenario  that  illustrated  potential  use  of  tools  in  all  of  the  tool 
areas.  The  scenario  was  based  on  a  hypothetical  6.3  (Advanced  Technology  Development  & 
Demonstration)  program  called  the  Federated  Data  Management  for  Air  Logistics  Centers 
(FDM/ALC).  It  contained  both  hardware  and  software  project  elements.  The  notion  was  that  the 
Air  Force  needed  portable,  laptop-computer-like  units  that  could  provide  flight-line  maintenance 
technicians  with  all  of  the  maintenance  information  they  required  for  a  given  aircraft,  in  “real 
time”  and  while  physically  on  the  flight  line.  The  scenario  contained  four  parts  designed  to 
illustrate  each  of  the  tool  areas.  It  was  implemented  using  an  World  Wide  Web  server  and 
browser  loaded  onto  a  laptop  PC. 

The  following  paragraphs  describe  the  content  of  each  of  the  presentations.  A  description  of  the 
scenario  for  each  tool  area  is  also  given. 


2.2  S&T  IPPD  Process  Overview  Presentation 

This  presentation  consisted  of  a  review  of  the  overall  S&T  IPPD  Process  and  motivation  for  its 
implementation.  Included  was  a  review  of  the  AL  Tools  and  Methods  task  within  the  initiative 
and  an  introduction  to  the  S&T  IPPD  process  model. 
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2.3  (Technology)  Requirements  Management  Presentation 

The  requirements  management  presentation  addressed  the  first  two  activities  of  the  S&T  IPPD 
process  model  (Define  Requirements  and  Establish  S&T  Exit  Criteria).  The  presentation  focused 
on  the  application  of  Quality  Function  Deployment  (QFD)  and  the  House  Of  Quality  (HOQ)  to 
S&T  needs. 

Requirements  Scenario.  The  requirements  management  portion  of  the  scenario  illustrated  how  a 
program  manager  could  respond  to  a  short  suspense  “crisis”  by  rapidly  tracking  down  and 
compiling  information  to  justify  key  program  decisions  that  were  made  earlier  in  the  program’s 
life,  prior  to  his  or  her  own  tenure  with  the  project.  This  portion  of  the  demonstration  made  use 
of  a  workflow-tool  interface  which  was  modeled  after  the  interface  to  Metro,  a  workflow  tool  by 
Action  Technologies.  It  also  demonstrated  an  HOQ  with  “animated”  call-up  windows 
(implemented  in  ActiveX).  It  did  not  illustrate  the  original  capture  and  organization  of  the 
requirements.  This  portion  of  the  scenario  was  the  most  effective  of  the  four  areas  for  two 
reasons:  (1)  participants  could  understand  in  very  concrete  terms  how  requirements  analysis 
tools  could  help  them,  and  (2)  they  could  directly  relate  the  scenario  to  their  own  S&T  programs. 


2.4  (Value  Judgment  via)  Group  Consensus  Presentation 

The  group  consensus  presentation  addressed  the  overall  need  for  consensus  activities  at  various 
points  throughout  the  S&T  IPPD  process,  as  well  as  an  approach  for  achieving  group  consensus. 

Group  Consensus  Scenario.  The  group  consensus  portion  of  the  scenario  illustrated  an  approach 
to  determine  which  laboratory  mission  priorities  should  be  addressed  by  the  FDM/ALC  project. 
The  technique  that  was  illustrated  is  called  Successive  Proportional  Additive  Numeration  or 
SPAN,  and  involves  the  “blind”  assignment  of  “points”  or  “votes”  among  the  decision-making 
participants  followed  by  a  voting  process.  This  portion  of  the  scenario  appeared  to  be  the  least 
effective  because,  to  the  research  team’s  surprise,  participants  had  trouble  relating  to  laboratory 
“mission  priorities”  and  there  appeared  to  be  some  concern  that  tools  and  techniques  such  as 
SPAN  might  dilute  program  manager  authority  and  his  or  her  ability  to  direct  the  effort.  (On  the 
contrary,  where  such  tools  have  been  implemented,  they  tend  to  reinforce  program  manager 
decisions  rather  than  dilute  program  manager  authority.) 


2.5  (Design)  Value  Analysis  Presentation 

The  value  analysis  presentation  addressed  activities  3  and  4  of  the  S&T  IPPD  Process 
(Determine  Technology  Alternatives  and  Perform  Value  Analysis).  This  area  was  the  most 
challenging  because  it  is  central  to  the  notion  of  applying  variability  metrics  to  determine  the 
relative  value  of  competing  technologies. 

Value  Analysis  Scenario.  In  this  portion  of  the  scenario,  the  participants  were  stepped  through 
an  illustrative  S&T  value  scorecard.  There  was  insufficient  time  during  the  interviews  to  bring 
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the  participants  down  to  the  level  of  the  design  worksheets,  from  which  the  numbers  in  the  value 
scorecard  usually  derive.  As  a  result,  there  was  some  angst  with  respect  to  how  the  numbers  in 
the  scorecard  would  be  generated  in  an  actual  program. 


2.6  (Program  Management)  Workflow  Presentation 

The  workflow  presentation  addressed  issues  in  the  underlying  workflow  infrastructure  required 
for  effective  management  of  complex  programs  with  distributed  Integrated  Product  Teams. 
(IPTs).  This  presentation  helped  participants  understand  the  differences  between  worl^ow, 
groupware,  and  group  consensus  tools,  as  explained  in  section  3.3. 

Workflow  Scenario.  The  IPPD  process  model’s  concept  of  workflow  and  interaction  with  a 
“control  center”  permeated  the  entire  scenario  demonstration.  This  concept  “stole  the  thunder” 
from  the  actual  workflow  demonstration.  Workflow  was  the  second  most  difficult  tool  area  for 
which  to  elicit  meaningful  requirements.  First,  participants  were  not  familiar  with  the  concept. 
Second,  workflow  unnecessarily  complicates  the  management  of  simple  programs  run  by  a  few 
collocated  people.  For  many  laboratory  programs,  it  would  be  “overkill.”  Third,  most  of  the 
participants  have  little  experience  on  programs  that  require  a  high  degree  of  integrated 
interaction,  including  automatic  tasking  and  tracking  among  members  of  a  distributed  IPT. 
Nevertheless,  many  participants  saw  significant  potential  value  for  workflow  tools,  but  viewed 
their  implementation  as  a  long-term  proposition. 


2.7  Effectiveness  of  Approach 

The  IPPD-process-overview  portion  and  the  requirements-management  portion  of  the  interview 
were  very  effective  at  communicating  the  tasking  that  S&T  managers  will  receive  during  IPPD. 
Participants  gained  a  sufficient  understanding  to  effectively  generate  and  prioritize  their  needs 
for  supporting  tools. 

The  value  analysis  portion  of  the  interview  provided  a  good  overview  of  this  tool  area. 
Participants  viewed  the  value  scorecard  as  a  good  thing,  but  the  necessary  details  were  too 
overwhelming  to  enable  effective  generation  of  user  needs.  A  positive  side  effect  of  the 
interview  was  that  many  participants  wanted  to  take  the  Design  for  Six  Sigma  Manufacturing 
(DSSMyCapstone  Course  and  devote  the  time  necessary  to  more  frilly  understand  this  area. 

The  group-consensus  portion  of  the  interview  never  worked  well.  The  user  needs  generated  by 
the  participants  were  based  primarily  on  the  features  and  capabilities  that  the  research  team  had 
provided.  The  participants  liked  the  tool  they  experienced  (i.e.,  Ventana  GroupSystems),  and 
many  thought  that  the  approach  was  effective  for  brainstorming  activities.  However,  the 
participants  did  not  understand  the  need  for  consensus  activities  in  the  S&T  IPPD  process 
outside  of  brainstorming.  The  research  team,  therefore,  concluded  that  many  of  the  techniques 
available  to  drive  a  group  toward  consensus  must  be  directly  experienced,  rather  than  briefed,  for 
comprehension. 
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The  workflow  portion  of  the  interview  was  another  area  that  was  too  complex  for  the  uninitiated 
to  sufficiently  grasp  to  delineate  their  needs  for  features  and  capabilities.  Most  participants  saw 
workflow  tools  simply  as  project  management  tools,  and  wanted  to  compare  workflow  to 
Microsoft  Project.  In  fact,  it  appears  that  tools  like  Microsoft  Project  are  a  step  up  from  what 
many  program  managers  currently  use. 
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3.  Results  by  Tool  Area 


3.1  (Technology)  Requirements  Management 

A  requirements-management  tool,  in  broad  terms,  provides  for  the  collection,  organization,  and 
analysis  of  customer-derived  requirements  for  a  project  and  technology  application.  Such  a  tool 
could  also  be  used  in  performing  correlation  analysis  between  customer  requirements  and 
engineering  design  criteria. 

Interview  participants  were  first  given  the  presentation  described  in  paragraph  2.3  covering  this 
tool  area.  This  briefing  served  as  a  high-level  introduction  to  the  capabilities  of  requirements 
management  tools,  the  categories  of  this  type  of  tool  (e.g.  Quality  Function  Deployment  (QFD), 
Systems  Engineering),  and  products  and  trends  in  tool  development. 

A  major  objective  of  the  interviews  was  to  query  the  participants  about  the  features  these  tools 
should  possess  in  order  to  assist  in  implementing  IPPD.  Preceding  formal  data  collection,  warm¬ 
up  questions  were  asked  of  the  group  to  stimulate  thoughts  about  their  needs  for  tool  capabilities. 
The  following  paragraphs  present  the  results  of  the  structured  interviews  and  the  tools  identified 
as  candidates  to  provide  requirements  management  capabilities.  The  tools  are  then  evaluated 
against  the  needs  identified  by  the  users  during  the  interviews.  Also  included  are  a  discussion  on 
market  trends  related  to  requirements  management  tools,  and  recommendations  for  future 
development  in  this  tool  area. 

3.1.1  Interview  Results 

The  majority  of  attendees  were  strongly  interested  in  the  requirements-management  tool.  The 
disjointed  and  difficult  task  of  capturing  and  managing  requirements  for  technology  development 
projects  can  be  overwhelming.  The  deployment  of  an  automated  tool  could  serve  to  ease  the 
process  and  allow  for  the  instant  generation  and  manipulation  of  data. 

A  set  of  warm-up  questions  was  developed  to  stimulate  the  participants’  thinking  about  the 
features  and  capabilities  of  requirements  management  tools.  During  the  successive  interviews, 
this  set  of  questions  evolved  based  on  participant  feedback  and  the  research  team’s  review  of  the 
answers  elicited.  The  final  set  of  questions  and  representative  responses  are  summarized  below. 

How  does  your  organization  capture  and  manage  project  requirements,  and  what  tools  and 
methods  have  proven  helpful? 

•  Projects  are  defined  through  bench  scientist  research  and  contracted  firont-end 
analysis. 

•  Typically,  by  identifying  potential  customers  and  holding  face-to-face  meetings. 
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•  In  general,  we  do  not  systematically  capture  and  manage  to  a  set  of  project 
requirements. 

•  We  use  ad-hoc  tools  that  are  rarely  beneficial  to  the  “gaining”  program. 

•  This  is  done  by  informal  group  consensus  among  those  involved  in  developing 
proposals  and  Statements  Of  Work  (SOWs).  While  the  process  is  generally 
extensive  and  well  done  by  the  time  it's  finished,  it  is  haphazard,  unintegrated,  and  not 
always  well  documented. 

•  Unfortunately,  few  tools  and  methods  have  proven  effective.  A  tool  like  the  one  we 
are  using  might  help  facilitate  reviewing  and  responding  to  user  requirements  with 
our  plans’  office. 

•  Mostly  manually!  Lists  of  requirements  versus  solutions  in  a  spreadsheet  is  the  most 
sophisticated  case.  Generally,  requirements  have  been  few  and  at  high  level,  vvith 
only  a  few  possible  technical  approaches  considered. 

•  Meeting  minutes,  people’s  notes,  red-lined  documents,  action  items,  etc. 

In  some  cases,  structured  techniques  were  noted  as  being  used  for  capturing  requirements.  These 
included  IDEF  modeling,  QFD,  site  visits,  electronic  mail  (e-mail),  interview  sessions.  Joint 
Application  Development  (JAD)  sessions,  brainstorming  sessions,  and  general  process 
approaches  (that  start  with  analyzing  a  set  of  determined  needs  and  end  with  an  established  list  of 
project  features,  constraints,  assertions,  and  priorities). 

Is  the  customer  integrated  into  the  process  of  gathering  requirements?  How? 

•  We  have  started  integrating  customers.  However,  there  is  no  formal  method. 

•  As  much  as  possible,  normally  through  the  Integrated  Product  Team  (IPT). 

•  There  is  no  way  to  define  requirements  without  the  customer.  (Lots  of  travel,  e-mail, 
fax,  and  video/telephone  conversation  are  used.) 

•  They  are  usually  invited  to  participate  in  the  Technology  Transition  IPT.  (They  may 
not  be  an  active  participant  -  but  may  review  what  is  generated.) 

•  If  the  customer  is  paying  for  the  activity,  s/he  may  be  more  pro-active. 

Was  the  scenario  helpful  or  not  helpful  in  understanding  requirements  management? 

Why? 

•  Yes.  You  can  see  how  to  access  the  information  you  need,  as  long  as  you  can 
maneuver  from  one  concept  to  the  next  without  getting  lost.  Always  have  that  big 
picture  available. 

•  Yes.  The  scenario  was  useful  and  I  hope  to  see  how  6.2  efforts  might  also  be 
addressed. 
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•  The  House  Of  Quality  (HOQ)  process  is  not  much  different  from  a  file  plan,  in  that  it 
is  only  as  good  as  the  discipline  followed  when  using  it.  If  the  discipline  isn't 
followed  carefully,  the  HOQ  is  essentially  worthless. 

•  One  of  the  most  interesting  aspects  of  the  tools  demonstrated  is  the  permanent 
capturing  of  decision  logic  and  supporting  studies.  This  sure  beats  digging  through 
file  cabinets  after  someone  moves  to  another  job! 

Would  the  demonstrated  approach  to  requirements  management  be  useful  to  you  in  your 
projects?  Please  explain, 

•  I'm  not  sure  my  staff  could  understand  the  complex  subject  matter.  I  also  was  looking 
for  the  Mission  Need  Statement  (MNS)  or  the  Operational  Requirements  Document 
(ORD)  developed  by  the  customer.  It  wasn't  in  the  scenario  but  should  have  been. 

•  Yes,  if  my  customers  were  more  directly  involved  in  stating  and  reviewing  their 
requirements.  Unfortunately,  most  requirement  statements  I  see  are  generated  and/or 
interpreted  by  headquarters  staffs.  Consequently,  I  spend  a  great  deal  of  time 
verifying  the  needs  statements  I  receive  from  my  plans  office  with  my  customers 
located  in  the  field. 

•  Yes.  This  is  a  formalization  of  the  design  process  that  is  usually  (always?)  done 
anyway,  and  would  be  useful  in  saving  time,  catching  gaps  and  errors  in  the  design, 
and  documenting  the  decisions. 

What  key  elements  of  requirements  definition  or  analysis  have  we  missed? 

•  The  mission  area  planning  (MAP)  process. 

•  What  is  needed  in  your  software  support  tool  for  the  HOQ  documentation  process  is 
something  to  highlight  everything  that  is  possibly  affected  by  every  change  that  is 
made  to  the  HOQ.  In  other  words,  everything  that  is  linked  to  a  single  location  on 
the  HOQ,  or  everything  that  is  linked  to  a  location  that  is  linked  to  the  location  where 
the  change  was  made,  or  to  a  location  that  is  linked  to  a  location  that  is  linked  to  a 
location ...  In  other  words,  you  have  to  be  able  to  see  the  total  impact  of  each  change 
that  is  made  to  the  HOQ.  This  is  what  often  fails  in  the  requirements  definition 
process.  People  lose  track  of  why  they  didn't  go  in  that  direction  in  the  first  place. 

•  You  have  missed  the  possibility  of  encountering  multiple  customer  groups  with 
conflicting  requirements.  We  need  assistance  in  reaching  consensus  among  them  to 
produce  a  final  requirements  priority  list. 

•  You  missed  the  operators’  requirements  documents  and  the  test  organization 
coordination. 
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From  these  participant  comments,  the  following  general  observations  can  be  made: 

1.  Requirements  management  in  6.3  programs  is  mostly  a  manual,  imstructured  process. 

2.  Program  managers  attempt  to  keep  customers  involved  but  generally  have  no  structured 
approach  to  do  so. 

3.  QFD  and  the  HOQ  are  viewed  as  productive  methodologies. 

4.  Program  managers  are  very  concerned  about  requirements  traceability,  linkage,  rationale, 
and  supporting  material. 

5.  Program  managers  are  interested  in  access  to  official  requirements  documents  generated 
by  customers  such  as  Mission  Need  Statements  (MNSs)  and  Operational  Requirements 
Documents  (ORDs). 

Bottom  Line:  These  comments  highlight  the  need  for  a  more  disciplined,  structured  approach  in 
managing  requirements  in  research  programs.  Furthermore,  a  tool  to  help  implement  such  a 
process  is  needed. 

The  information  gathered  during  this  question-and-answer  session  set  the  stage  for  the  next  step 
in  the  interview  process  --  building  a  list  of  desired  capabilities  for  a  requirements  management 
support  tool. 

3.1.2  User  Needs  for  Requirements-Management  Tools 

In  the  first  interview,  a  blank  page  was  provided  in  the  GroupSystems  software  for  the 
interviewees  to  enter  features  and  capabilities  desired  in  a  requirements-management  tool.  In  the 
rest  of  the  sessions,  it  was  determined  a  more  useful  approach  would  be  to  provide  the 
participants  with  a  pre-established  list  of  features  (-with  definitions)  for  their  comment  and 
refinement.  The  pre-defined  list  of  features  and  capabilities  for  a  requirements  collection, 
organization,  and  analysis  tool  was: 

1.  Requirements  Definition,  Decomposition,  and  Allocation:  The  tool  should  provide 
significant  automation  support  to  the  process  of  creating,  storing  and  displaying  original  and 
derived  project  requirements  from  top-level  user  requirements  down  to  detailed  design  and 
performance  requirements.  This  includes  the  ability  to  designate  requirement  categories,  and 
to  link  requirements  to  other  types  of  data  elements/objects  such  as  verification  methods. 

The  tool  should  provide  the  capability  to  provide  rationale  for  requirements. 

2.  Requirements  Traceability:  The  tool  should  make  it  easy  to  follow  linked  requirements 
(or  other  objects  or  elements)  to  parents,  children  or  siblings,  providing  a  clear,  multilevel 
view  of  their  traceability.  Graphical  depiction  is  preferable. 

3.  Requirements  History:  The  tool  should  maintain  the  change  histories  for  all  database 
elements/objects,  preferably  automatically  as  the  changes  are  made.  The  tool  should  also 
provide  the  ability  to  capture  rationale  for  the  changes. 
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4.  Identification  and  Tracking  of  Program  Issues:  The  tool  should  pennit  the  input  of 
programmatic  issues  and  the  association  of  those  issues  with  the  appropriate  system  elements 
(e.g.,  requirement,  function,  interface,  etc.).  It  also  should  provide  a  mechanism  for 
specifying  the  status  of  each  issue,  capturing  the  eventual  resolution  of  each  issue,  and 
capturing  the  effects  of  that  resolution,  such  as  additional  or  amended  requirements. 

5.  System  Architecture  Definition:  The  tool  should  support  the  definition  of  system 
functions,  interfaces,  and  components.  It  should  provide  the  means  to  allocate  functions  to 
system  components.  It  should  provide  a  hierarchy  or  other  graphical  representation  of  the 
component  structure. 

6.  Standard  Document/Query  Generation:  The  tool  should  have  the  capability  to  easily 
generate,  on  demand,  hardcopies  of  standard  reports  (e.  g.,  database  listings,  traceability 
reports,  orphan/widow  reports,  etc.)  and  queries.  The  ability  to  create  or  import  graphs  and 
tables  is  desired.  A  variety  of  standard  output  formats  is  also  desired,  including  ASCII  and 
Microsoft  RTF. 

7.  Custom  Document/Query  Generation:  The  tool  should  allow  creation  of  custom 
reports  such  as  individual  traceability  reports,  element  description  reports,  custom  queries, 
etc.,  'without  specialized  tool  knowledge  or  training.  Word-processor  style  functions  are 
desired.  The  tool  should  provide  the  ability  to  create  templates  for  use  when  creating 
standard  documents. 

8.  User  Interface:  The  tool  should  have  a  user  interface  that  is  intuitively  easy  to  learn  and 
operate.  A  Graphical  User  Interface  (GUI)  is  preferable  to  a  command  line  approach.  The 
tool  should  support  a  wide  variety  of  users. 

9.  Bridges  to  Other  Support  Tools:  The  tool  shoidd  have  well-defined,  trouble-free 
interfaces  to  other  commercial  products,  particularly  the  Microsoft  Office  suite  of  office 
support  tools.  It  also  should  provide  links  to  analysis  tools,  groupware,  and  the  worldwide 
web  (WWW).  It  should  provide  an  ability  to  use  multimedia. 

10.  Analysis:  The  tool  should  provide  the  capabilities  to  conduct  trade  studies,  compare 
competing  solutions,  examine  impacts  of  changes,  verify  test  acceptance  criteria,  and 
prioritize  requirements. 

11.  Configuration  Management:  The  tool  should  have  configuration  management 
capabilities  that  allow  establishment  and  control  of  baselines  and  versions  of  both 
requirements  and  documents.  The  tool  also  should  provide  mechanisms  to  route,  track, 
coordinate,  and  incorporate  changes  to  documents. 

12.  Reuse:  The  tool  should  provide  the  capability  to  reuse  requirements  sets,  supporting 
documentation,  etc.  once  they  have  been  entered  into  the  database.  For  example,  if  a 
requirements  set  has  been  established  for  a  system  or  subsystem,  this  set  could  be  used  in  the 
requirements  identification  of  another  system  or  subsystem  without  re-entering  the  data. 

13.  Interfaces  to  Other  Activities:  The  tool  should  provide  interfaces  to  other  activities 
such  as  acquisition,  procurement,  etc. 
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14.  Multi-User  Support:  The  tool  should  support  simultaneous  database  access  by  multiple 
users  and  provide  the  means  to  coordinate  database  changes  to  avoid  conflicts. 

15.  Access  Control:  The  tool  should  provide  password  protection  and  allow  administrator 
assignment  of  read  and  write  privileges  to  any  portion  of  the  main  database  for  each  user. 

16.  Security:  The  tool  should  provide  some  level  of  built-in  support  for  databases  in  secure 
environments,  such  as  automated  security  markings  on  displays  and  hard  copies. 

17.  Cost: 

•  Initial  Cost  -  Cost  of  software  license,  documentation,  installation,  and  required 
support  resources, 

•  Training  Cost  -  Cost  of  any  necessary  vendor-conducted  user  training,  and 

•  Recurring  Cost  -  Yearly  cost  to  keep  license  current  and  receive  software  updates. 

18.  Performance  as  Database  Grows  Large:  The  tool  should  accommodate  large 
databases  (>10,000  objects  or  records)  with  little  or  no  performance  degradation.  Is  there  a 
practical  limit  beyond  which  the  tool  may  become  imusable? 

19.  Platforms  Supported:  The  tool  should  be  hosted  on  Windows,  Macintosh,  and  UNIX 
machines. 

20.  Vendor  Support:  Vendor  support  should  be  accessible  and  provides  timely,  helpful 
answers  to  tool  implementation  problems.  The  vendor  should  offer  a  range  of  application 
support  capabilities. 

21.  Maturity  Indicators: 

•  The  length  of  time  the  tool  has  been  in  the  marketplace, 

•  The  number  of  companies  using  the  tool  to  support  real  programs  including 
commercial  programs,  and 

•  The  extent  to  which  there  have  been  known  problems  with  the  tool. 

22.  Database  Tailorability/Extendibility:  The  tool  should  support  tailoring  and  extension 
of  the  existing  database  structure  to  better  conform  to  particular  program  terminologies  and 
methodologies.  Such  tailoring  and  extension  should  not  result  in  loss  of  any  tool 
functionality. 

23.  Tool  User  Documentation  /Training:  The  vendor  should  provide  quality 
documentation  with  the  tool.  User  manuals  should  be  comprehensive,  imderstandable,  and 
include  tutorials  for  beginning  users. 


24.  System  Modeling:  The  tool  should  provide  the  capability  to  create  and  execute  the 
functional,  data,  and  control  flows  of  a  system  and  its  subsystems. 
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25.  Capability  to  Show  Process/Demonstrate  Requirements:  The  tool  should  provide  the 
capability  to  simulate  the  process  or  requirement  that  will  be  addressed.  The  simulation 
could  be  compared  to  the  current  environment  in  order  to  demonstrate  potential 
improvements,  cost  savings,  and  process  changes. 

26.  Requirements  Sensitivity  Analysis:  The  tool  should  allow  exploration  of  “what  if’ 
scenarios  with  the  requirements  and  their  weightings  to  determine  how  the  program  may 
change  with  downstream  requirements.  The  tool  should  save  these  'what  ifs'  separately  from 
the  official  set  of  requirements. 

The  participants  were  given  the  opportunity  to  comment  on  each  of  the  above  listed  features  and 
capabilities.  They  were  also  allowed  to  add  or  delete  features  and  capabilities.  Each  interview 
arrived  at  a  list  of  25  to  30  features  that  the  selected  tool  should  have  within  its  functional 
capacity.  When  the  participants  had  finished  editing  and  commenting  on  the  list,  a  vote  was 
taken  to  determine  which  features  were  most  important.  The  voting  results  for  all  five  interviews 
are  shown  in  table  2.  Desired  features  are  lined  up  across  the  table  so  that  it  is  easy  to  see  the 
degree  of  consensus.  Note  that  only  the  top  twelve  choices  from  each  interview  are  shown. 

These  are  designated  as  “major”  features. 

The  voting  results  shown  in  Table  2  reveal  that  there  is  a  core  set  of  features  upon  which  most  of 
the  participants  agreed  for  a  requirements  management  tool.  For  the  purposes  of  this  analysis,  a 
feature  was  considered  as  a  “core”  feature  if  it  were  rated  as  major  a  feature  (i.e.,  ranked  in  the 
top  12)  at  least  three  times.  Given  this  definition,  the  core  features  identified  for  a  requirements 
management  tool  are: 

1.  Analysis 

2.  Configuration  Management, 

3.  Identification  and  Tracking  of  Program  Issues, 

4.  Multi-User  Support, 

5.  Requirements  Decomposition,  Definition,  Allocation, 

6.  Requirements  Sensitivity  Analysis, 

7.  Requirements  Traceability, 

8.  Standard  Document/Query  Generation,  and 

9.  User  Interface. 

The  results  also  show  that  some  features  were  only  important  to  one  of  the  participant  groups. 
This  supports  the  research  team’s  belief  that  every  program  will  have  different  needs  based  on 
the  maturity  of  its  technology  and  on  the  type  of  research  involved. 
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3.1.3  Market  Assessment  for  Requirements-Management  Tools 

The  following  paragraphs  summarize  of  the  market  assessment  for  requirements-management 
tools.  First,  the  major  features  of  interest  are  reviewed.  Then,  a  summary  is  given  of  the  tools 
available  and  how  each  tool  supports  the  major  features  of  interest.  Next  follows  a  discussion  of 
market  trends  in  terms  of  industry  initiatives.  Finally,  recommendations  are  made  for  future 
enhancements  of  requirements-management  tools. 

3.1.3.1  Major  Features  of  Interest 

The  major  features  of  interest  are  those  features  that  were  ranked  in  the  top  12  features  at  least 
one  time  during  the  interviews.  Considering  a  feature  major  even  if  it  was  ranked  in  the  top 
twelve  features  only  one  time  supports  the  research  team’s  belief  that  all  programs  are  different 
and  may  require  unique  capabilities.  These  features  became  the  focus  of  interest  in  the  market 
assessment.  They  are: 

1 .  Access  Control, 

2.  Analysis, 

3.  Captures  Expertise, 

4.  Configuration  Management, 

5 .  Identification  and  Tracking  of  Program  Issues, 

6.  Interfaces  to  Other  Tools, 

7.  Multi-User  Support, 

8.  Platforms  Supported:  Macintosh, 

9.  Platforms  Supported;  UNIX, 

10.  Platforms  Supported:  Windows, 

1 1 .  Prioritization  Capability, 

12.  Requirements  Definition,  Decomposition,  and  Allocation, 

13.  Requirements  History, 

14.  Requirements  Sensitivity  Analysis, 

15.  Requirements  Traceability, 

16.  Reuse, 

17.  Risk  Management, 

18.  Standard  Document/Query  Generation, 

19.  System  Architecture  Definition, 

20.  Tool  User  Documentation  /Training, 

21.  User  Interface,  and 

22.  Vendor  Support. 
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3.1.3.2  Requirements-Management  Tool  Candidates 

The  market  assessment  included  identifying  tools  that  could  potentially  support  the  features  and 
capabilities  listed  above.  Research  of  the  market  for  requirements  collection,  organization,  and 
analysis  tools  showed  that  the  demand  for  and  supply  of  these  tools  has  increased  dramatically  in 
the  last  decade.  Although  systems  engineering  as  a  methodology  has  been  around  since  the 
1950’s,  only  recently  have  computer  technology  advancements  made  it  possible  to  provide  the 
storage  capacities  and  processing  speeds  necessary  for  automating  systems  engineering.  Since 
the  computer  power  is  now  available,  software  vendors  have  been  offering  more  tools.  Among 
the  emerging  automated  tools  are  specialized  requirements-management  tools.  These  tools 
concentrate  on  capturing  and  managing  requirements  and  producing  requirements  specifications. 
The  focus  of  this  tools  search  was  to  identify  automated  tools  (either  QFD  or  systems 
engineering  tools)  that  support  requirements  management.  The  results  of  the  search  showed  that 
automated  tools  are  available  to  support  both  the  QFD  approach  and  the  systems  engineering 
process.  Once  identified,  these  tools  could  be  evaluated  in  terms  of  how  they  satisfy  the  major 
features  of  interest  shown  in  paragraph  3. 1.3.1  above.  The  tools  identified  as  potential 
candidates  for  requirements  management  are  shown  in  table  3  along  with  relevant  administrative 
information. 

As  shown  by  table  3,  the  market  offers  several  requirements-management  tools.  Also,  note  that 
the  tools  vary  in  price  from  $950  to  $50,000.  This  wide  range  in  price  is  indicative  of  the 
robustness  of  the  tools,  as  will  be  seen  in  the  next  paragraph. 

3.1.3.3  Requirements-Management  Tool  Evaluations 

Using  the  needs  identified  for  a  requirements-management  tool  and  the  list  of  COTS  tools 
identified  as  potential  candidates,  an  analysis  was  conducted  to  see  which  tools  satisfied  the 
needs  identified  during  the  interviews.  No  recommendation  was  made  regarding  which  tools  a 
program  manager  should  use.  This  decision  must  be  made  by  the  program  manager  based  on 
his/her  program’s  particular  needs. 


Table  3.  Requirements-Managemeiit  Tool  Candidates 


Gpinpanj/ 

Product 

Pescriptiod 

Pnce 

Contact 

Qualisoft 

QFD  Designer 

PC  Based. 
Automates  QFD 
Methodology 

$975 

4652  Patrick  Rd. 

West  Bloomfield,  MI  48322 
(810)645-2561 

International 

TechneGroup 

Inc. 

QFD/CAPTURE 

PC  Based. 
Automates  QFD 
Methodology 

$950 

(800)783-9199 

Quality 
Systems  & 
Software 

DOORS 

(Dynamic  Object 
Oriented 
Requirements 
System) 

Requirements 

Management 

$5,000 

11921  Freedom  Dr. 

Reston,  VA  22090 
(703)904-4360 
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Tables.  (Concluded) 


mmmm 

;  Descriptioii 

Price 

Contact  ^ 

Marconi 

Systems 

Technology 

RTM 

(Requirements 
Traceability  and 
Management ) 

Requirements 

Management 

$20,000 

1861  Wiehle  Ave. 

Reston,  VA  22090 
(703)736-3525 

Teknowledge 

Corporation 

ProductTrack 

Requirements 

Management 

$50,000 

10  user 
licenses 

1810  Embarcadero  Rd 

Palo  Alto,  CA  94303 
(415)424-0500 

TD 

Technologies 

SLATE 
(System  Level 
Automation  Tool 
for  Engineers) 

Requirements 

Management 

$10,000  to 
$18,000 

6140  Parkland  Blvd. 

Ma5dield  Heights,  OH 

44142 

(216)460-4700 

Mesa 

Systems 

Guild 

Cradle  SEE 

Requirements 

Management 

$14,000 

60  Quaker  Lane 

Warwick,  RI  02886 
(401)828-8500 

Vitech 

Corporation 

CORE 

Requirements 

Management 

$8,000 

■ 

2070  Chain  Bridge  Rd 

Suite  105 

Vienna,  VA  22182 
(703)883-2270 

Ascent  Logic 
Corporation 

RDD-100 

Requirements 

Management 

$12,000 

180  Rose  Orchard  Way 

San  Jose,  CA  95032 
(408)943-0630 

Compliance 

Automation, 

Inc. 

VITAL  LINK 

Requirements 

Management 

$5,690 

17629  El  Camino  Real 

Suite  207 

Houston,  TX  77058 
(713)486-7817 

Teledyne 

Brown 

Engineering 

Xtie-RT 

Requirements 

Management 

$6,000 
first  seat, 
$1,499 
each  add. 

300  Sparkman  Dr.  NW 

P.O.  Box  070007 

Huntsville,  AL  35807 

Requisite, 

Inc. 

Requisite 

Requirements 

Management 

$795 

4720  Table  Mesa  Dr. 

Boulder,  CO  80303 
(303)499-9177 

Armstrong 

Laboratory 

Requirements 
Analysis  Process 
in  Design- 
Weapon  Systems 
(RAPID-WS) 

Requirements 

Management 

N/A 

AL/HRGA 

WPAFB,  OH  45433 
(513)255-8502 

Evaluations  of  the  requirements-management  tools  were  accomplished  by  reviewing  vendor 
literature  and  by  reviewing  other  tool  reviews  by  third  parties  such  as  the  International  Coxmcil 
on  Systems  Engineering  (INCOSE)  and  the  Aerospace  Corporation.  In  some  cases, 
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demonstration  copies  of  the  tools  were  obtained  and  reviewed.  In  other  cases,  vendors 
demonstrated  the  tools.  In  two  cases  (SLATE  and  RAPID-WS),  hands-on  use  of  the  tools  was 
possible. 

The  methods  of  tool  evaluation  are  important  for  two  reasons.  First,  the  “vaporware”  syndrome 
prevalent  today  may  cause  vendors  to  exaggerate  their  products’  features  beyond  their  true 
capabilities.  Second  and  perhaps  more  importantly,  an  evaluation  of  software  based  only  on 
literature  and  demonstrations  may  not  be  as  thorough  as  it  should  be.  Even  though  the  literature 
contains  words  and  language  the  reviewer  thinks  he/she  understands,  the  possibility  exists  for 
miscommvmication.  Two  outcomes  of  a  misconununication  are  possible.  One  is  that  the 
reviewer  believes  the  tool  provides  a  certain  feature  when  it  doesn’t.  The  other  is  the  reviewer 
believes  the  tool  doesn’t  provide  a  certain  feature  when  it  does.  Either  outcome  should  be 
mitigated  by  further  evaluation  via  hands-on  use. 

The  results  of  the  evaluations  are  presented  in  table  4.  Each  candidate  tool  was  evaluated  against 
the  major  features  identified  in  the  interviews.  The  table  should  be  interpreted  as  follows: 

1 .  •  indicates  the  tool  offers  strong  support  for  the  feature, 

2.  >  indicates  the  tool  offers  medium  support  for  the  feature, 

3.  O  indicates  the  tool  offers  only  limited  or  weak  support  for  the  feature,  and 

4.  A  blank  cell  indicates  the  tool  offers  no  support  for  the  feature. 

Program  managers  need  to  remember  the  imcertainty  associated  with  the  evaluation  techniques 
discussed  above  when  using  table  4  to  choose  a  requirements-management  tool.  Since  the 
interviews  supported  the  research  team’s  belief  that  all  programs  have  unique  needs,  the 
approach  of  identifying  candidate  tools  for  further  hands-on  exploration  by  each  program  is 
preferable  to  that  of  recommending  one  tool  for  all  programs. 

3.1.4  Market  Trends  in  Requirements-Management  Tools 

As  mentioned  above,  more  vendors  are  offering  requirements-management  tools.  As  the  tools 
become  more  sophisticated,  many  companies  are  offering  modular  products.  For  example,  many 
of  the  high-end  systems-engineering-tool  vendors  offer  their  products  in  stand-alone  modules 
that  separate  the  functions  of  systems  engineering  into  requirements  management,  modeling, 
systems  design/engineering,  and  document  management.  This  separation  can  be  beneficial  to 
those  users  who  are  only  interested  in  particular  functions  such  as  requirements  management. 

The  opportunity  to  buy  additional  modules  as  they  are  needed  is  also  attractive. 
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Table  4.  Requirements-Management  Tool  Evaluations 
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Level  of  Support:  •  =  Strong,  I  =  Medium,  O  =  Weak,  Blank  =  None 
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Level  of  Support:  •  =  Strong,  I  =  Medium,  O  =  Weak,  Blank  =  None 


Another  trend  observed  is  the  demand  for  these  types  of  tools  to  interface  with  other  tools  such 
as  Microsoft  Office,  other  analysis  tools,  etc.  Vendors  are  responding  to  this  demand  by  building 
Object  Linking  and  Embedding  (OLE),  Open  Data  Base  Connectivity  (ODBC),  and 
import/export  capabilities  into  their  products.  Many  of  these  tools  also  offer  Application 
Programming  Interfaces  (APIs)  which,  in  some  cases,  give  the  users  broad  capabilities  to 
customize  and  tailor  their  tools. 

It  must  be  mentioned  that  the  ability  of  a  tool  to  operate  in  a  heterogeneous  environment  is  fast 
becoming  a  prerequisite  with  users.  In  fact,  Microsoft  Windows  NT  is  becoming  the  server  of 
choice,  joining  Novell  and  UNIX  as  a  leader  in  network  management.  As  long  as  users  have  the 
ability  to  choose  between  these  operating  systems,  the  need  for  platform-independent 
applications  will  continue. 

3.1.5  Recommendations  for  Future  Development 

Considering  the  findings  of  the  interviews,  it  is  clear  that  program  managers  need  requirements 
management  tools.  From  the  market  assessment,  it  appears  that  several  tools  are  viable 
candidates.  The  current  assessment  of  requirements  management  tools  revealed  the  following 
fundamental  shortfalls  in  today’s  applications: 

•  World-Wide  Web  (Internet)  enabled  capabilities  are  not  sufficiently  rich. 

•  Security  issues  at  all  levels,  including  access  and  encryption,  have  not  been  adequately 
addressed. 

•  Risk  management  capabilities  in  these  tools  are  virtually  nonexistent. 

•  Interfaces  to  other  tools  are  lacking. 

Some  of  these  issues,  such  as  Web-enabled  capabilities  and  security,  are  being  addressed  by 
software  vendors.  Any  future  enhancements  and  deployment  of  these  tools  should  address  these 
shortfalls. 

The  following  steps  outline  an  approach  to  providing  a  tool  for  use  in  requirements  management. 

1 .  Select  a  program  for  which  to  choose  and  deploy  a  tool.  The  obvious  choice  is  one  of  the 
S&T  IPPD  pilot  programs. 

2.  Choose  one  of  the  candidate  tools  that  provides  the  core  features  as  described  in 
paragraph  3.1.2. 

3.  Modify  the  tool.  Modification  falls  into  two  categories — ^modifications  for  the  unique 
features  that  the  pilot  program  needs  and  modifications  to  address  the  shortfalls  of  current 
tools  as  described  above.  Close  interaction  with  the  program  manager  and  the  IPT  will  be 
required  to  ensure  that  all  needs  are  met.  (Perhaps  meetings  using  a  group  consensus  tool 
would  be  in  order  to  facilitate  this  interaction.) 
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4.  Customize  the  tool  to  encompass  the  requirements  and  test  documents  (e.g.,  MNS  and 
ORD)  that  the  program  requires. 

5.  Develop  a  test  plan.  Deploy  the  tool  for  testing  over  an  appropriate  time  period. 

6.  Use  lessons-leamed  and  user  feedback  to  improve  the  deployments  to  the  rest  of  the  pilot 
programs  and,  ultimately,  the  entire  S&T  commimity. 

In  terms  of  interfaces  to  other  tools,  it  is  recommended  that  an  interface  be  built  between  the 
requirements-management  tool  and  a  group-consensus  tool.  This  interface  should  allow  two-way 
data  exchange  so  that  requirements  can  be  exported  to  the  group  consensus  tool  for  analysis  and 
imported  back  into  the  requirements  tool  after  completion.  It  must  be  noted  that  such  an 
interface  has  been  prototyped  by  AL/HRGA  on  the  RAPID-WS  research  program.  (The 
interface  was  built  between  RAPID-WS  and  GroupSystems.)  This  work  should  be  leveraged  for 
any  future  development  efforts  on  S&T  IPPD  Tools  and  Methods. 


Given  the  mature  nature  of  the  market  for  requirements  tools  and  the  work  already  accomplished 
on  the  RAPID-WS  program,  the  amoimt  of  time  needed  to  customize  and  deploy  this  kind  of  tool 
should  be  minimal.  Specifically,  most  of  the  features  and  capabilities  wanted  in  this  kind  of  tool 
are  already  provided  by  several  of  the  tools  as  shown  in  table  4.  Thus,  the  recommended  timing 
and  level  of  effort  for  deploying  requirements  tools  are  as  follows: 


Person-Hours 

Deliverables 


FY97 

3000 

Customized  tool 
deployed  for  one 
pilot  program 


FY98 

5000 

Customized  tool 
deployed  for 
three  more  pilot 
programs 


FY99 

6000 

Customized  tool 
deployed  for 
remaining  pilot 
programs 


FYOO 

2000 

Customization 
and  Deployment 
Plan  for  S&T 
community 


FY01 

TBD 

Customization 
and  deployment 
to  entire  S&T 
community 


3.2  (Value  Judgment  via)  Group  Consensus 

Following  the  established  interview  protocol,  a  group  consensus  tool  presentation  was  given  as 
an  introduction  to  this  tool  area.  This  served  as  a  high-level  introduction  to  how  this  category  of 
tool  could  be  used  within  the  context  of  the  S&T  IPPD  process  model,  the  capabilities  of  this 
type  of  tool,  the  general  categories  of  this  type  of  tool  (e.g.,  collaboration,  communication),  and 
trends  in  product/tool  development. 

General  comments  on  the  introduction  to  group-consensus  tools  showed  that  this  kind  of  tool 
would  be  valuable  in  achieving  consensus  for  various  pilot  projects,  especially  given  the  variety 
of  experts  that  would  participate  in  the  process.  Most  participants  agreed  that  consensus  is  a 
powerful  tool  in  that  it  builds  on  peoples’  strengths,  not  weaknesses. 

The  remainder  of  this  section  presents  the  results  of  the  interview  sessions  and  the  tools 
identified  as  candidates  to  provide  group-consensus  capabilities.  The  tools  are  then  evaluated 
beised  on  the  needs  identified  by  the  users  during  the  interviews.  Also  included  are  a  discussion 
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on  market  trends  related  to  group-consensus  tools  and  recommendations  for  future  development 
in  this  tool  area. 

3.2.1  Interview  Results 

As  was  the  case  for  a  requirements-management  tool;  the  majority  of  interviewees  agreed  that  a 
group  consensus  tool  would  be  of  great  help  in  managing  their  programs.  The  program  manager 
frequently  faces  the  difficult  task  of  reaching  consensus  for  projects  in  the  6.3  technology  arena. 
The  deployment  of  an  automated  tool  could  ease  the  collaborative  process,  facilitate 
communication  flow,  and  enable  IPTs  to  agree  on  project  terms,  requirements,  design,  etc. 

A  set  of  “warm-up”  questions  was  developed  to  stimulate  the  participants’  thinking  about  group 
consensus  tools.  As  was  the  case  for  requirements-management  tools,  this  set  of  questions 
evolved  based  on  participant  feedback  and  the  research  team’s  review  of  the  answers  elicited. 
The  final  set  of  questions  and  representative  responses  are  summarized  below. 

What  tools  and  methods  do  you  use  to  reach  consensus  and  how  do  you  employ  them? 

•  Besides  face-to-face  discussion,  we  use  only  the  telephone  and  electromail. 

•  IPTs  are  developed  before  contract  award.  They  employ  meetings,  fax,  mail,  and 
e-mail  for  coordination. 

•  I  use  brainstorming,  affinity  techniques,  nominal  group  techniques,  inter¬ 
relationship  diagram  analysis,  and  prioritization  techniques. 

•  Usually  the  strongest  personalities  within  the  IPT  carry  the  day. 

•  We  use  all  types  of  tools:  utility  analysis,  multiple  criteria/options,  weightings, 
and  pairwise  comparison. 

•  Consensus  occurs  ad  hoc  over  several  years  -  during  which  the  project  refines  its 
scope  and  definition. 

•  These  tools  seem  to  be  a  substitute  for  a  program  manager  who  doesn't  use  some 
sort  of  internal  decision  process,  or  know  the  participants  in  the  process.  They,  in 
essence,  dump  all  that  responsibility  on  the  group.  In  most  applications,  this 
could  easily  become  a  crutch  and  mask  more  serious  experience  or  ability 
problems  within  management. 

•  They  cannot  replace  the  program  manager.  Regardless  of  the  tools,  ultimately 
someone  must  make  the  decisions.  They  are  tools  only  to  help  the  program 
manager.  Sometimes  a  program  manager  may  have  a  preconceived  solution  and 
go  with  it  despite  the  team. 

•  We  use  no  automated  tools  to  reach  consensus. 
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What  are  the  benefits  and/or  shortcomings  of  using  group-consensus  tools  to  support 
both  distributed  and  co-located  IPTs? 

•  Consensus  gives  assurance  that  a  big  mistake  isn’t  being  made  or  an  opportunity, 
lost. 

•  Consensus  allows  your  stakeholders  more  voice  in  the  process  and  provides  an 
active  vehicle  for  commitment. 

•  Consensus  tools  that  provide  for  distributed  collaboration  are  fast,  but  the  U.S. 
Mail  system  works  --  it’s  just  slower.  Also  Federal  Express  or  its  clones  work 
well.  I  have  used  all. 

•  A  “Bimch  Of  Guys  Sitting  Around  Talking”  (BOGS  AT)  is  used  too  much  in  AF 
plaiming. 

•  This  could  be  an  effective  way  of  coordinating  IPT  direction. 

•  Shortcoming:  Hidden  agendas  and  not  hearing  answers/opinions  prior  to 
distributing  votes  is  a  concern.  Benefit:  Hidden  way  of  allowing  for  group 
empowerment  and  relying  on  experts. 

•  The  benefit  is  a  coordinated  initial  requirement.  Shortcomings  are  not  in  the  tools 
themselves,  but  in  their  application  (i.e.,  are  all  stakeholders  participants  in 
negotiations?). 

Did  the  overview  enable  you  to  evaluate  the  requirement  for  group  consensus  tools  to 
support  the  S&  T  JPPD  process? 

•  It  is  obvious  that  such  tools  would  be  beneficial,  but  as  far  as  understanding  the 
key  features  that  would  be  required  to  implement  successful  tools,  no.  Maybe  we 
went  through  that  part  too  quickly. 

•  I  think  this  part  was  covered  quickly.  If  substantially  more  time  was  spent  on 
every  survey  portion  like  this  one,  this  interview  would  last  more  than  two  days. 
Nevertheless,  more  time  would  have  to  be  spent  understanding  the  IPPD  process 
and  the  group-consensus  scenario,  and  in  weighing  available  capabilities,  for  us  to 
understand  our  requirements  for  group-consensus  tools. 
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•  It  would  have  been  more  productive  if  a  discussion  had  followed  the  briefing,  or 
taken  place  during  the  briefing. 

•  Not  really.  Unlike  the  QFD  scenario,  there  were  no  examples.  I  only  have  a 
notion  of  what  some  tools  are  like  and  how  they  would  compare  with  one  another 
—  advantages  and  disadvantages  —  when  applied  to  different  projects. 

•  The  overview  was  a  good  introduction  to  possible  tools  that  are  available,  and  to 
the  capabilities  they  could  bring  to  the  decision-making  process. 

•  Yes.  It  was  a  good  introduction.  Of  course,  it  did  not  recommend  a  specific  tool 
for  our  use. 
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What  key  elements  of  group  consensus  have  we  missed? 

•  Conflict  resolution  -  identifying  conflicts,  their  solutions  when  possible,  and 
impasses  that  must  be  addressed  in  the  future. 

•  The  human  element  needs  to  be  addressed  in  more  detail:  how  to  control 
outspoken  participants,  identify  the  true  experts  versus  the  charlatans  and  snake 
oil  salesmen,  keep  hidden  agendas  from  derailing  the  process,  etc. 

From  these  participant  comments,  the  following  general  observations  can  be  made: 

1 .  The  methods  that  program  managers  now  use  to  reach  group  consensus  range  from 
purely  ad  hoc  procedures  to  Total  Quality  Management  (TQM)  techniques. 

2.  Reaching  consensus  is  an  ongoing  activity  throughout  a  project’s  existence. 

3.  There  is  concern  that  these  types  of  tools  may  usurp  the  responsibilities  of  program 
managers. 

4.  Automated  tools  are  not  presently  used  to  reach  consensus. 

5.  The  benefits  of  consensus  tools,  as  well  as  some  shortcomings,  were  recognized. 

Bottom  Line:  These  comments  point  to  the  need  for  a  tool  to  help  IPTs  reach  consensus  on 
various  issues.  Although  in  many  cases  sound  techniques  are  being  used,  in  many  more, 

BOGS  AT  is  still  practiced.  The  remarks  also  stress  the  importance  of  proper  utilization  of  such  a 
tool.  Specifically,  the  tool  can  never  replace  the  program  manager,  Avith  whom  the  ultimate 
responsibility  of  a  program  rests.  Such  tools,  however,  can  assist  program  managers  in  making 
and  justifying  better  decisions. 

The  information  gathered  during  this  question-and-answer  session  set  the  stage  for  the  next  step 
in  the  interview  process  —  building  a  list  of  tool  needs. 

3.2.2  User  Needs  for  Group  Consensus  Tools 

As  for  the  requirements-management  tool  area,  the  participants  for  this  tool  area  were  provided  a 
pre-established  list  of  features  for  their  comment  and  refinement.  Each  session  arrived  at  a  fairly 
consistent  list  of  approximately  20  consensus  tool  functional  features.  The  predefined  list  of 
features  and  capabilities  for  a  group-consensus  tool  was: 

1.  Easy  to  Use:  The  tool’s  interface  should  be  user-friendly.  A  GUI  is  preferable  to 
command  line  approach.  The  tool  should  be  easy  to  install  and  maintain.  The  tool  should 
have  import/export  capabilities  to  other  tools  such  as  word  processors  and  spreadsheets. 

2.  Anytime,  Anywhere:  The  tool  should  be  capable  of  handling  collaboration  activities 
without  regard  to  the  times  they  occur  or  the  location  of  the  participants.  The  anywhere 
condition  promotes  "virtual”,  real-time  meetings.  The  anytime  condition  allows  users  to 
access  data  and  information  whenever  they  are  able  to  do  so,  but  not  necessarily  in  real  time. 
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3.  Active  Notification:  The  tool  should  notify  participants  whenever  new  information  has  been 
entered  into  the  session,  allowing  immediate  review. 

4.  Anonymous  User  Input:  The  tool  should  allow  group  members  the  option  of  entering 
their  thoughts  anonymously.  Anonymous  input  promotes  the  sharing  of  more  ideas.  This 
feature  is  especially  useful  when  brainstorming. 

5.  Brainstorming:  The  tool  should  enable  all  participants  to  express  their  ideas  and 
thoughts  on  the  subject  matter  being  explored.  Additionally,  the  tool  should  allow  further 
discussion  and  clarification.  For  example,  the  tool  should  give  participants  the  ability  to  add 
annotations,  comments,  or  questions  about  comments  entered  by  other  participants. 

6.  Voting:  The  tool  should  provide  a  voting  mechanism.  Voting  can  be  used  to  see  if  a 
group  is  in  consensus  on  issues.  For  example,  if  the  group  has  delineated  the  requirements 
for  a  project,  the  voting  mechanism  could  be  used  to  prioritize  ("rack  and  stack")  the 
requirements.  If  the  variance  is  large  on  some  of  the  requirements,  the  group  can  revisit 
those  issues  to  attempt  resolution. 

7.  Report/Document  Generation:  The  tool  should  provide  the  capability  to  capture  the 
entire  electronic  group-consensus  session  and  then  export  the  information  to  other  tools  (e.g., 
a  word  processor).  By  using  the  other  features  in  this  list,  teams  could  use  this  type  of  tool  to 
outline  reports/studies,  agree  on  content,  assign  responsibilities  to  team  members,  etc. 

8.  Structured  Decision  Analysis:  The  ability  to  reach  consensus  on  subjective  and 
objective  issues  is  vital  to  an  IPT.  A  consensus  tool  should  provide  the  capability  to  use 
structured  decision  analysis  techniques,  either  as  part  of  the  tool  or  as  an  add-on.  For 
example,  the  tool  should  allow  the  IPT  to  list,  categorize,  and  prioritize  issues/topics. 

Further,  the  tool  should  allow  the  IPT  to  use  techniques  such  as  the  Analytic  Hierarchy 
Process  (AHP)  or  value  engineering  methodologies  to  reach  consensus  on  these  issues/topics. 

9.  Facfiitator/Moderator  Functionality:  The  key  to  successful  group-consensus  meetings 
is  facilitation.  The  tool  should  provide  the  facilitator  with  centralized  control  of  the  activities 
in  which  the  participants  engage. 

10.  Categorizer:  As  mentioned  in  structured  decision  analysis,  the  tool  should  provide  the 
capability  to  categorize  issues  and  topics.  For  example,  during  a  brainstorming  session,  the 
group  may  list  100  requirements  for  a  new  aircraft.  It  would  probably  be  useful  to  categorize 
the  requirements  in  terms  of  the  major  aircraft  subsystems  that  address  them  -  e.g.,  airframe, 
avionics,  and  propulsion. 

11.  Prioritizer:  The  tool  should  provide  the  capability  to  prioritize  information.  For 
example,  after  an  IPT  has  listed  and  categorized  requirements  for  a  new  aircraft,  the  team 
may  want  to  prioritize  the  requirements  in  terms  of  "must  have"  or  "nice  to  have." 

12.  Multimedia  Input:  The  capability  to  incorporate  all  types  of  data  (audio,  video,  text, 
graphics)  is  desirable.  Furthermore,  the  capability  to  conduct  “anywhere”  meetings  through 
audio  and  video  capabilities  is  desirable.  Market  trends  indicate  these  capabilities  are  being 
developed  and  should  be  commonplace  within  five  years. 
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13.  Whiteboard:  The  tools  should  be  capable  of  providing  interactive,  real-time 
information  sharing  via  a  whiteboard  facility.  This  capability  is  especially  useful  when 
conducting  “anywhere”  meetings. 

14.  Concept  Visualization  Techniques:  The  tool  should  provide  the  capability  to  visually 
demonstrate  a  concept,  process,  or  product  using  rapid  prototyping,  simulations,  or  quick- 
and-dirty  software  demonstrations. 

15.  History:  Group  interaction  should  be  threaded  to  track  ideas  or  comments  through 
ongoing  communications.  One  should  be  able  to  see  the  original  comment  and  all 
appropriate  responses  in  one  seamless  area,  without,  for  example,  needing  to  open  seven 
messages  to  get  to  an  idea. 

The  participants  were  given  the  opportunity  to  comment  on  each  of  the  above  listed  features  and 
capabilities.  They  were  also  allowed  to  add  or  delete  features  and  capabilities.  Each  interview 
arrived  at  a  list  of  15  to  20  features  that  the  selected  tool  should  have  within  its  functional 
capacity.  When  the  participants  had  finished  editing  the  list,  a  vote  was  taken  to  determine 
which  features  were  most  important.  The  voting  results  for  all  five  interviews  are  shown  in 
Table  5.  As  with  our  list  of  features  for  the  requirements-management  tools;  only  the  top  12 
choices  from  each  interview  are  shown  as  major  features.  Desired  features  are  lined  up  across 
the  table  so  that  it  is  easy  to  see  the  degree  of  consensus. 

The  voting  results  shown  in  table  5  reveal  that  there  is  a  core  set  of  features  upon  which  most  of 
the  participants  agree.  For  the  purposes  of  this  analysis,  a  feature  was  considered  as  a  core 
feature  if  it  was  ranked  as  a  major  feature  (in  the  top  1 2)  at  least  three  times.  Given  this 
definition,  the  core  features  identified  for  a  group-consensus  tool  are: 

1 .  Action  Item  T racking, 

2.  Active  Notification, 

3.  Anytime,  Anywhere, 

4.  Brainstorming, 

5.  Concept  Visualization  Techniques, 

6.  Easy  to  Use, 

7.  Facilitator/Moderator  Functionality, 

8.  Prioritizer, 

9.  Report/Document  Generation, 

10.  Structured  Decision  Analysis,  and 

11.  Voting. 

As  was  the  case  for  requirements-management  tools,  the  results  show  that  some  features  were 
important  to  only  one  of  the  participant  groups.  This  again  supports  the  research  team’s  belief 
that  every  program  will  have  different  needs  based  its  level  of  maturity  and  on  the  type  of 
research  involved. 
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iteboard  Whiteboan 


3.2.3  Market  Assessment  for  Group-Consensus  Tools 

The  following  paragraphs  summarize  of  the  market  assessment  for  group-consensus  tools.  First, 
the  major  features  of  interest  are  reviewed.  Then,  a  summary  is  given  of  the  tools  available  and 
how  each  tool  provides  the  major  features  of  interest.  Next,  a  discussion  of  market  trends  is 
presented.  Finally,  recommendations  are  made  for  future  enhancements  of  group-consensus 
tools. 

3. 2.3.1  Major  Features  of  Interest 

The  major  features  of  interest  were  ranked  in  the  top  12  in  at  least  one  interview  session.  These 
features  became  the  focus  of  interest  in  the  market  assessment.  They  are: 

1 .  Action  Item  T racking, 

2.  Active  Notification, 

3.  Anonymous  User  Input, 

4.  Anytime,  Anywhere, 

5.  Brainstorming, 

6.  Calendar  or  Scheduling  Tool, 

7.  Categorizer, 

8.  Concept  Visualization  Techniques, 

9.  Easy  to  Use, 

10.  Facilitator/Moderator  Functionality, 

1 1 .  History, 

12.  Import/Export, 

13.  Platforms  Supported:  Macintosh, 

14.  Platforms  Supported:  UNIX, 

15.  Platforms  Supported:  Windows, 

16.  Prioritizer, 

17.  Report/Document  Generation, 

18.  Sensitivity  Analysis, 

19.  Structured  Decision  Analysis, 

20.  Voting,  and 

21.  Whiteboard. 
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3.2.3.2  Group-Consensus  Tool  Candidates 

Groupware,  as  a  concept,  has  been  around  for  a  long  time.  Its  current  definition  is:  “Groupware 
is  not  only  the  forms  or  processes  we  use  to  shape  our  interactions,  but  it  is  the  capacity  to  create, 
shape,  and  change  these  forms  and  processes  as  appropriate.”  Groupware  did  not  exist  as  an 
accepted  software  category  until  Lotus  Notes  was  first  released  in  1989. 

Because  the  definition  of  groupware  is  so  broad,  products  that  fall  into  this  include: 

•  Electronic  Mail  (e-mail):  This  category  allows  the  most  basic  type  of  group 
interaction  -  messaging  --  and  generally  includes  the  ability  to  attach  files. 

•  Workgroup-Enabled  Applications:  These  types  of  applications  offer  built-in  file¬ 
sharing  features. 

•  Scheduling:  These  tools  allow  workgroup  members  interactive  access  to  information 
about  meetings  and  events.  Some  tools  include  their  own  e-mail  capabilities  while 
others  are  meant  to  work  in  conjimction  with  existing  e-mail  systems. 

•  Conferencing:  These  relatively  new  applications  allow  users  in  geographically 
separate  locations  to  share  information  on-screen  in  real-time  (to  support  “anywhere” 
meetings). 

An  integral  part  of  group-consensus  functionality  is  conferencing.  Conferencing  products,  thus, 
became  the  focus  of  this  segment  of  our  research. 

Our  search  for  COTS  tools  that  offer  conferencing  fimctionality  supported  the  notion  that  the 
conferencing-software  market  is  expanding  rapidly.  Table  6  overviews  the  potential  candidates 
for  group  consensus  tools.  Once  identified,  these  tools  were  evaluated  against  the  major  features 
listed  in  paragraph  3.2.3. 1. 


Table  6.  Group  Consensus  Tool  Candidates 


ji 

Contact 

Ventana 

Corporation 

GroupSystems 

PC  Based. 
Collaboration, 
an3^ime, 
anywhere 

$895 

user 

1430  E.  Ft.  Lowell  Rd 

Suite  301 

Tucson,  AZ  85719 
(800)368-8319 

TRAX  Softworks 
Inc. 

TeamTalk  2.0 

PC  Based. 
Collaboration, 
anytime, 
anywhere 

$395 

5  users 

5840  Uplander  Way 

Culver  City,  CA 

90230-6620 

(800)367-8729 

Enterprise 
Solutions,  Inc. 

MeetingWorks 
for  Windows 

PC  Based. 
Collaboration, 
anytime, 
anywhere 

Free 

60  Union  St. 

Suite  3232 

Seattle,  WA  98101 
(206)467-1234 
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Table  6.  (Concluded) 


lEorapahy 

Product 

Description 

Cost 

Contact 

Expert  Choice 

Inc. 

Team  Expert 
Choice  for 
Windows 

Decision 

Support,  AHP 

$14,295 

Software 

and 

Hardware 

5001  Baum  Blvd. 

Suite  650 

Pittsburgh,  PA  15213 
(412)682-3844 

TeamWARE 

TeamWARE 

Collaboration, 

anytime, 

anywhere 

$2,495, 

10  seats 

800  Central  Expressway 
Santa  Clara,  CA  95052 
(408)982-9143 

Crosswise 

Corporation 

FacetoFace 

Collaboration, 

anytime, 

anywhere 

$59 

1 05  Locust  St. 

Santa  Cruz,  CA  95060 

Data  Fellows  Ltd. 

Vineyard 

Collaboration, 

anytime, 

anywhere 

$295  ea. 
$18,000 
for  100 

users 

4000  Moorpark  Ave. 

Suite  207 

San  Jose,  CA  95117 
(408)244-9090 

White  Pine 

CU-SeeMe 

Collaboration, 

anytime, 

anywhere 

$295 

542  Amherst  St. 

Nashua,  NH  03063 
(603)886-9050 

Collabra 

Software  Inc. 

Collabra  Share 
1.01 

Collaboration, 

anytime, 

anywhere 

$995 

server, 

$99  user 

1091  N.  Shoreline 

Mountain  View,  CA  94043 
(800)474-7427 

Attachmate 

OpenMind  1.0 

Collaboration, 

anytime, 

anywhere 

$995 

server, 

$295 

user 

8230  Montgomery  Rd. 
Cincinnati,  OH  45236 
(513)794-8290 

Mesa  Group,  Inc. 

Conference+  1.1 

Collaboration, 

anytime, 

anywhere 

$75 

29  Crafts  St. 

Newton,  MA  02160 
(617)964-7400 

Avantos 

Performance 

DecideRight  1.0 
for  Windows 

Decision 

Support 

$149 

5900  Hollis  St. 

Emer5wille,  CA  94608 

3.2.3.3  Group-Consensus  Tool  Evaluations 

The  same  tool  evaluation  approach  was  used  for  group-consensus  tools  as  was  used  for 
requirements-management  tools.  With  the  needs  for  a  group-consensus  tool  delineated  and  the 
potential  COTS  tool  candidates  identified,  the  two  were  compared.  Again,  no  recommendation 
was  made  regarding  specific  tools  a  program  manager  should  use. 

The  results  of  the  tool  evaluations  are  presented  in  table  7.  As  with  the  requirements- 
management  tools,  each  candidate  group-consensus  tool  was  evaluated  against  the  features 
identified  in  the  interviews.  Only  the  top  twelve  features  fi'om  each  interview  were  evaluated. 
The  table  should  be  interpreted  as  before: 
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1 .  •  indicates  the  tool  offers  strong  support  for  the  feature, 

2.  >  indicates  the  tool  offers  medium  support  for  the  feature, 

3.  O  indicates  the  tool  offers  only  limited  or  weak  support  for  the  featvure,  and 

4.  A  blank  cell  indicates  the  tool  offers  no  support  for  the  feature. 

As  with  the  requirements-management-tool  evaluations,  program  managers  should  use  the  results 
of  the  group-consensus  evaluations  to  choose  tools  that  appear  to  provide  the  features  they  need, 
then  evaluate  the  tools  for  themselves. 

3.2.4  Market  Trends  in  Group  Consensus  Tools 

The  future  for  groupware,  particularly  conferencing/consensus  products,  appears  to  be  promising. 
As  the  trend  towards  workgroups  and  the  team  model  increases,  more  applications  such  as  word 
processors  and  spreadsheets  will  integrate  workgroup  functionality.  More  focused  tools,  such  as 
the  Analytic  Hierarchy  Process  (AHP)  for  decision  analysis,  are  also  gaining  popularity. 

Ovum,  Ltd.  attributes  19  percent  of  the  present  groupware  market  to  stand-alone  e-mail  systems 
and  33  percent  to  workflow  systems.  Ovum  forecasts  that,  as  e-mail  functionality  is  integrated 
with  other  technologies,  stand-alone  e-mail  will  account  for  only  10  percent  of  the  market  in 
1998. 

The  scheduling  and  conferencing  groupware  market  segments  are  expanding  rapidly,  as  new 
vendors  add  products  with  increasing  frequency  and  users  buy  them  to  fill  their  needs. 
WorkGroup  Technologies  expects  scheduling-tool  revenue  to  grow  by  103  percent  in  1997,  while 
conferencing-tool  revenue  will  increase  by  171  percent  during  the  same  period. 

The  Internet  will  also  impact  the  groupware  market.  The  ability  to  conference  over  the  World- 
Wide  Web  (“Web”)  holds  great  promise.  In  fact,  some  of  the  tools  identified  in  this  research  are 
already  Web-enabled.  Although  the  Web  is  exploding  rapidly,  it  has  not  yet  become  the 
“information  superhighway”  envisioned.  As  it  grows  and  matures,  it  is  likely  to  have  great 
impact  on  workgroup  tools.  For  this  reason,  it  is  important  to  stay  informed  about  the  advances 
in  Web  technology  and  its  impact  on  group-consensus  tools.  This  is  not  to  say  that  tools  that  are 
not  Web-enabled  cannot  support  group  consensus.  However,  it  may  be  the  case  that  Web- 
enabled  products  can  support  group  consensus  in  more  ways. 
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Table  7.  Group  Consensus  Tool  Evaluation 
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Level  of  Support:  •  =  Strong,  I  =  Medium,  O  =  Weak,  Blank  =  None 


Table?.  (Concluded) 
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Level  of  Support:  •  =  Strong,  I  =  Medium,  O  =  Weak,  Blank  =  None 


3.2.5  Recommendations  for  Future  Development 

Considering  the  findings  of  the  interviews,  it  is  clear  that  program  managers  need  group- 
consensus  tools.  From  the  market  assessment,  it  appears  that  several  tools  are  viable  candidates. 
The  current  assessment  of  group-consensus  tools  revealed  the  following  fundamental  shortfalls 
in  today’s  applications: 

•  Web  (Internet)  enabled  capabilities  are  not  sufficiently  rich. 

•  Security  issues  at  all  levels,  including  access  and  encryption,  have  not  been 
adequately  addressed. 

•  Import/export  capabilities  are  not  supported. 

•  Sensitivity  analysis  is  not  provided. 

•  Interfaces  to  other  tools  are  lacking. 

Several  of  the  tools  shown  in  Table  7  provide  most  of  the  features  that  users  need. 
Recommendations  for  future  development  in  this  tool  area  parallel  those  made  for  the 
requirements-management  tool  area.  Because  these  areas  are  closely  interrelated,  the  same  pilot 
program  should  be  chosen.  A  group-consensus  tool  should  be  chosen  that  provides  those  core 
features  identified  in  paragraph  3.2.2.  Because  group-consensus  tools  and  requirements- 
management  tools  have  similar  shortcomings,  many  of  the  modifications  desired  for  each  will 
also  be  similar.  It  is  recommended,  however,  that  a  tool  be  chosen  that  is  Web-enabled,  to 
enhance  distributed  IPT  collaboration.  The  same  test  and  deployment  procedures  should  be 
carried  out  as  outlined  for  the  requirements-management  tool.  Again,  it  is  emphasized  that  the 
work  accomplished  on  the  RAPID-WS  program  must  be  leveraged  in  any  effort  to  customize 
these  tools. 


The  schedule  and  level  of  effort  appropriate  for  adopting  group-consensus  tools  should  be  similar 
to  those  appropriate  for  adopting  requirements-management  tools. 


FY97  FY98  FY99  FYOO  FY01 

Person-Hours  2000  4000  6000  1000  TBD 


Deliverables 


Customized  tool 
deployed  for  one 
pilot  program 


Customized  tool 
deployed  for 
three  pilot 
programs 


Customized  tool 
deployed  for 
remaining  pilot 
programs 


Customization 
and  Deployment 
Plan  for  S&T 
community 


Customization 
and  deployment 
to  entire  S&T 
community 


3.3  (Program  Management)  Workflow 

“Workflow”  is  a  relatively  new  term  and,  as  such,  is  generally  not  well  understood.  Articles  on 
workflow  paint  sometimes  conflicting  pictures  and  often  mingle  terms  such  as  “workflow”  and 
“groupware”.  As  figure  1  suggests,  workflow  and  groupware  play  related  but  play  different  roles 
in  the  business  process. 
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“Workflow,”  first,  can  refer  to  a  collection  of  tasks  or  activities  in  a  business  process.  Second,  it 
can  refer  to  the  execution  of  those  tasks  and  activities.  Third,  it  can  refer  to  the  software  used  to 
manage,  measure,  track,  coordinate,  and  revise  those  tasks  and  activities.  In  the  context  of  this 
effort,  “workflow”  was  used  to  represent  a  software  environment  or  infrastructure  that  supports 
multiple  workers  and  applications  throughout  a  business  process. 


What  is  the  difference  between  groupware 
and  workflow?  “Groupware”  represents  the 
underlying  infrastructure  upon  which 
workflow  and  other  applications  (e.g.  group 
consensus  and  decision  support)  rely.  It  can 
be  thought  of  as  a  data  and  communications 
infrastructure,  while  workflow,  which  is  a 
layer  “above”  groupware,  can  be  thought  of 
as  the  business  process  control  infrastructure. 

Applications  such  as  group-consensus  tools 
can  be  implemented  in  groupware  systems, 
while  they  may  or  may  not  interact  with  a 
workflow  system. 

Note  that  we  speak  here  in  terms  of  groupware  and  workflow  systems.  The  workflow  systems  to 
which  we  refer  are  implemented  in  software,  due  to  the  complexity  of  today’s  information¬ 
intensive  environments  in  which  such  tools  are  being  employed.  Historically,  the  “flow  of  work” 
has  not  been  managed  in  any  area  other  than  manufacturing.  In  paper-based  organizations, 
everyone  learned  by  an  informal  apprenticeship  how  things  were  done,  what  went  where,  and 
how  long  things  took.  Organizations  depended  on  secretaries.  (A  few  still  do,  although  even  in 
those  cases,  the  role  of  the  secretary  is  changing,  and  the  ratio  of  secretaries  to  managers  has 
decreased  dramatically.)  The  secretaries  had  the  broadest  view  of  the  flow  of  work.  They  knew 
the  time  required  by  various  tasks  and  where  the  bottlenecks  existed. 

Modem  workflow  is  gradually  replacing  analogous  secretarial  functions.  It  enables  the  inclusion 
of  increased  functionality  in  the  business  process.  It  does  not  replace  human  review  and 
management  of  work,  but  it  substantially  augments  them.  Workflow  does  not  necessarily  mean 
that  business  processes  are  managed  or  performed  with  fewer  people.  It  does  mean  that  the 
processes,  which  are  now  more  flexible  and  responsive  to  changes  in  the  environment,  are 
tractable;  prior  to  the  advent  of  workflow,  they  were  not.  Workflow  is  replacing  certain  older 
functionality  (e.g.,  work  tracking  and  some  other  secretarial  functions),  integrating  common 
functionality  (e.g.,  scheduling),  enabling  more  complex  functionality  (e.g.,  task  negotiation  and 
process  analysis),  providing  for  the  management  of  a  vast  amount  of  information  and  control 
(decision  history,  progress  reporting,  distributed  management,  et  al.),  and,  ultimately,  supporting 
business  process  and  systems  engineering  with  improved  change-management. 

As  indicated  earlier,  it  was  necessary  to  provide  the  interviewees  with  considerable  background 
information  and  an  example  scenario  for  this  tool  area.  The  scenario  walked  them  through  a 


Figure  1.  Relationship  Between  Groupware, 
Workflow,  and  Group  Consensus 
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“day  in  the  life”  of  a  6.3  program  manager,  illustrating  how  a  workflow  system  could  ease  many 
critical  tasks.  Key  workflow  tasks  included: 

•  Project  sharing  and  allocation  among  personnel  and  resources; 

•  Event  management  to  enhance  a  proactive  capability  to  anticipate  pending  actions  and 
track  actions  in  progress; 

•  Notification,  request,  and  negotiation  of  tasking,  so  that  the  right  resources  are 
applied  to  various  tasks,  and  there  is  consolidated  notification  to  the  program  manager 
concerning  pending  and/or  overdue  actions; 

•  Integration  of  tasks  and  applications,  so  that  the  right  applications  are  applied  to  given 
tasks; 

•  Corporate  “memory”  via  the  active  capture,  organization  and  archival  of  the  process 
activities  and  information; 

•  Process  management  through  enforcement  of  the  business  rules;  and 

•  Continuous  process  improvement  by  enabling  the  effective  capture  of  process 
innovations  and  the  measurement  of  process  effectiveness. 

The  objective  of  the  interviews  was  to  introduce  participants  to  the  AFMC  S&T  IPPD  process 
and  to  collect  from  them  the  features  or  capabilities  that  these  tools  should  possess  in  order  to 
assist  them  in  implementing  the  process.  Following  the  briefings  and  scenario  demonstration/ 
simulation,  participants  were  asked  to  respond  to  a  set  of  warm-up  questions.  After  the  warm-up 
questions,  they  were  asked  to  review  and  help  build  a  list  of  functional  needs  that  workflow  tools 
should  support.  Finally,  they  were  asked  to  rank  those  functions  and  features  in  order  of 
importance  from  their  own  perspective  as  S&T  program  managers. 

3.3.1  Interview  Results 

As  expected,  participants  struggled  more  with  workflow  (and  value  analysis)  than  they  did  with 
the  requirements-management  or  group-consensus  portions  of  the  interviews.  The  reason  for 
their  difficulty  was  that  the  implementation  of  workflow  tools  is  not  a  familiar  activity,  and 
indeed,  the  workflow  market  is  still  defining  itself.  People  do  not  yet  employ  workflow  tools  the 
way  they  use  requirements-management  and  even  group-consensus  tools.  In  part  because  it  is 
more  difficult  to  understand,  workflow  was  viewed  as  a  potentially  critical,  but  longer  term, 
implementation  issue.  The  scenario  demonstrated  how  workflow  could  help  in  the  conduct  and 
management  of  a  complex  project.  Most  participants  indicated  they  would  like  to  have  access  to 
such  a  system.  However,  there  was  concern,  expressed  by  both  government  and  contractor 
personnel,  about  how  to  sufficiently  populate  a  workflow  system  to  make  it  really  useful  to  a 
program  manager. 

The  results  of  the  warm-up  questions  are  provided  below.  As  noted  earlier,  these  questions  did 
evolve  somewhat  during  the  course  of  the  interviews.  In  particular,  questions  with  respect  to  the 
effectiveness  of  the  scenario  demonstrations  were  added  for  the  third  through  fifth  interviews. 
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How  might  a  workflow  tool  enable  you  to  track  long-term  success  (Le.,  technology 
transfer)? 

•  It  would  provide  a  historical  database:  who  contributed  what,  when,  and  where;  what 
problems  occurred  and  how  they  were  corrected;  shortfalls;  pitfalls;  applications;  etc. 

•  I  believe  this  tool  would  be  helpful  but  not  to  the  same  level  as  some  of  your  earlier 
tools.  I  like  the  idea  of  this  tool  since  it  would  allow  me  better  visibility  into  how  the 
different  work  packages  are  progressing,  if/how  I  need  to  take  action,  and  how  my 
project  may  impact  other  projects. 

•  It  would  help  to  keep  all  the  project  information  intact.  Right  now  I'm  sure  there  are 
some  efforts  that  have  gone  through  several  years  of  work  with  several  different 
program  managers;  a  lot  of  the  documents  developed  and  decisions  made  are  nowhere 
to  be  found.  This  tool  could  enable  us  to  electronically  capture  management  history 
and  help  us  in  the  present. 

•  It  would  facilitate  communication  between  IPT  members  to  show  them  how  user 
requirements  are  being  translated  into  a  lab  prototype  and  how  that  is  being  brought 
to  EMD.  It  would  also  help  ensure  that  little  gets  lost  in  the  translation  from  one 
project  phase  to  another. 

•  This  question  is  a  little  disingenuous.  It's  pretty  obvious  that  tools  like  this  could 
keep  tilings  on  track,  alert  the  manager  to  problems  with  enough  lead  time  to  take 
corrective  action,  etc.  There'd  be  no  surprises  without  an  outright  effort  to  deceive. 

•  A  workflow  tool  would  allow  each  member  of  a  team  to  impact  decisions,  solutions, 
etc.  Thus,  the  process  and  thinking  that  went  into  the  decision  process  would  be 
maintained  and  it  would  be  easier  to  answer  why  or  why  not  you  considered  a 
particular  solution  or  technology.  In  addition,  it  would  provide  the  ability  to  look 
back  at  other  alternatives  and  pick  up  parts  of  them  that  may  help  solve  current 
problems. 

•  As  with  all  such  systems,  quality  consolidation  of  the  information  and  good  indexing 
for  retrieval  will  drive  the  overall  usefulness  of  tire  archived  information. 

How  might  workflow  tools  impact  the  execution  of  the  S&  T IPPD  process? 

•  I  don't  see  how  the  S&T  IPPD  process  can  be  implemented  effectively  without 
workflow  tools  that  work. 

•  These  tools  should  help  by  integrating  management  activities  into  one  environment. 

•  I  agree  with  the  above  statement,  and  would  add  that  tools  of  this  sort  could  help 
management  activities  in  general,  not  just  IPPD.  They  would  have  great  value  even  if 
there  were  no  IPPD  concept.  What  I  saw  covered  every  aspect  (that  I  can  think  of  so 
far)  of  oversight,  coordination,  etc.,  that  managers  need  to  do. 
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•  Workflow  tools  should  allow  the  process  of  technology  development  to  be  smoother 
and  more  efficient.  They  could  help  to  avoid  the  problems  of  schedule  conflicts  when 
everyone  can  not  get  together.  People  could  work  at  their  own  pace  as  long  as  they 
met  the  overall  schedule.  And,  if  a  member  of  a  group  were  only  interested  in 
impacting  one  or  two  parts  of  a  project,  he  could  look  at  only  them.  He  would  not 
have  to  waste  time  sitting  through  meetings  when  the  discussions  are  on  other 
subjects. 

•  They  could  enable  the  tracking  of  schedule  deviations,  identify  technology  challenges 
not  anticipated  during  original  plaiming,  help  resolve  issues  between  IPT  members, 
and  provide  a  simplified  means  of  information  reporting. 

•  They  are  critical  for  real  communication  between  distributed  teams  and  for  tracking 
whose  belly  button  needs  pushing.  Metrics  of  both  defined  and  ad-hoc  processes  are 
required  for  the  management  (and  I  think  all)  levels. 

•  Government  -  S&T,  Industry  -  S&T,  and  Industry  EMD  would  all  be  singing  from  the 
same  sheet  of  music.  This  means  two  things:  1)  S&T  would  be  less  intangible,  2) 
technology  transition  would  be  more  straightforward  (e.g.,  you  could  plug  in  another 
contractor  at  any  time  you  wished). 

Could  a  workflow  tool  help  build  advocacy  or  "buy  in"?  Please  Explain. 

•  This  is  a  tougher  question  than  the  others.  I'm  not  sure  how  it  would  help  buy-in  at 
the  start.  I  guess  as  things  went  along  and  some  of  the  risky  things  needed  to  get 
solved,  the  openness  of  the  tool  would  allow  everyone  the  same  understanding  of  the 
problems,  a  voice  in  fixing  them,  and  a  sense  of  ownership  due  to  having  helped. 

•  Would  management,  the  customer,  and  the  program  manager  all  have  unrestricted 
access  to  the  program  manager's  files?  I'm  not  sure  that  kind  of  exposure  is  good  for 
a  project.  It  would  not  give  the  program  manager  time  to  work  through  problems  on 
his/her  own,  and  would  invite  micromanagement.  The  program  manager  would  lose 
control. 

•  A  workflow  tool  would  support  buy-in  by  giving  all  of  the  team  members  access  to 
all  of  the  information  available  on  the  program.  Information  is  the  key  to  survival. 

•  A  workflow  tool  would  allow  all  parties  to  actively  participate  in  all  issues  at  any 
time.  It  would  allow  immediate  responses  to  questions  from  all  parties  involved. 

You  could  assume  this  would  promote  advocacy.  However,  the  program  manager 
would  then  be  responsible  for  assuring  all  parties  report  when  required  or  desired. 

•  "Buy  in"  must  extend  to  laboratory-management  and  contractor  Avillingness  to 
support  such  a  tool,  and  to  participate  (e.g.,  update  reports)  as  required.  I  think  that 
starting  simple  to  demonstrate  the  benefits  of  the  workflow  tool  would  help  build 
advocacy  within  the  laboratory,  laboratory  management,  and  even  other  contractors. 
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•  Two  ways:  First,  it  would  keep  users  plugged  in  to  the  program  throughout  its  life, 
which  helps  guarantee  some  technical  pull  from  the  users  at  the  worker  bee  level.  It 
also  would  help  when  pitching  to  decision  makers  (MAJCOMs,  SPOs,  etc.);  the 
program  manager  could  show  a  historical  perspective  of  the  program  as  part  of  a 
familiarization  briefing. 

•  It  could,  by  providing  a  medium  through  which  multi-party  dialogue  (from  users, 
investors,  sellers,  buyers,  etc.)  could  occur,  driving  toward  the  development  by  each 
party  of  a  personal  picture  of  what’s  in  it  for  them. 

Did  the  demonstration  enable  you  to  understand  workflow  tools  and  how  they  might 
support  the  S&T IPPD process ? 

•  I  think  the  demonstration  enabled  me  to  imderstand  the  use  of  workflow  tools.  I  think 
they  would  be  used  by  contractors  in  the  execution  of  ISCP  projects.  But,  I  still  find 
it  hard  to  see  how  they  could  be  implemented  within  the  laboratory. 

•  I  suspect  that  every  scientist  would  want  every  feature  that  has  been  described  here.  I 
also  suspect  that  if  anyone  bothered  to  survey  the  actual  use  of  the  features  among 
scientists  five  years  down  the  road,  they'd  find  that  very  few  of  these  features  were 
actually  used.  They  are  all  "nice  to  have",  but  I  don't  see  anyone  jumping  up  and 
down  saying  that  he  just  has  to  get  them  right  now. 

•  It  made  me  realize  that,  as  with  all  management  tools,  the  user  must  define  what  his 
requirements  truly  are  and  understand  the  options  available  before  he  can  make  a 
viable  tool  choice. 

•  Yes,  the  demonstration  allowed  me  to  understand  workflow’s  importance  to  the 
process.  However,  the  demonstration  also  scared  me.  There  are  many  tools  about 
which  I  have  no  working  knowledge.  There  will  be  no  delivered  tool  that  performs 
all  the  tasks  as  described.  Once  the  technic^  support  for  the  tool  is  gone,  who  is 
responsible?  Me.  I  would  probably  require  a  person  to  continue  this  support,  freeing 
me  to  handle  technical  and  program  issues. 

What  key  elements  of  workflow  have  we  missed? 

•  There  probably  are  some,  but  I'd  need  to  work  with  a  tool  to  find  them  by  realizing 
that  something  wasn’t  supported.  It  looks  pretty  thorough  conceptually. 

•  I  would  have  liked  to  have  seen  how  the  financial  functions  could  be  interfaced  into 
the  workflow. 

•  One  word:  Cost!  I  know  cost  is  based  on  level  of  implementation,  but  you  should 
give  team  members  some  idea  of  the  cost  for  these  tools.  S&T  projects  are  usually 
financially  constrained. 
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Based  on  these  comments,  the  following  general  observations  can  be  made: 


1 .  A  key  perceived  benefit  of  workflow  is  the  ability  to  track  project/corporate  history 
and  provide  an  audit  trail  of  the  decision  process. 

2.  The  capture  of  project  history  is  required  for  the  ability  to  manage  long-term 
programs  in  the  face  of  personnel  turnover,  and  for  the  transition  from  one  project 
phase  to  another. 

3.  An  important  function  of  workflow  is  to  facilitate  closer  linkage  and  increased 
involvement  (and,  thus,  ownership)  of  each  IPT  member. 

4.  There  is  no  consensus  yet  on  the  importance  of  workflow  in  S&T.  Some  individuals 
feel  it  is  essential  while  others  believe  it  is  a  longer  term  issue  for  which  the  payoff 
might  be  somewhat  dubious  today. 

5.  Program  managers  liked  the  idea  that  workflow  could  help  with  automatic 
notification  and  action  tracking  throughout  a  project,  but  they  were  concerned  about  a 
loss  of  control  over  the  project  should  upper  management  be  permitted  to  gain 
premature  access  to  project  activities.  At  issue  was  the  notion  of  the  management 
processes  to  control  access  to  worl^ow  systems. 

6.  The  scenario  demonstration  proved  to  be  an  extremely  important  aid  to  helping  the 
participants  visualize  how  various  tools,  especially  workflow,  could  support  S&T 
project  management.  This  fact  became  clear  when  comparing  the  quality  of  the 
feedback  from  the  first  two  sessions,  where  there  was  no  scenario,  with  that  of  the  last 
three  sessions,  where  the  scenario  was  added. 

7.  The  financial/cost  issue  was  not  sufficiently  addressed  in  the  scenario.  Comments  on 
cost  involved  four  areas:  the  use  of  workflow  to  support  cost  and  schedule  tracking 
throughout  a  project,  the  integration  of  workflow  with  applications  that  support 
transition  cost  estimation  during  S&T,  the  cost  of  implementing  workflow  in  fiscally 
constrained  S&T  projects,  and  the  cost  of  sustaining  workflow  software  in  a 
manpower-constrained  environment. 

Bottom  Line:  Workflow  is  potentially  very  important  to  S&T,  but  its  implementation  is  a  longer 
term  issue  that  depends  on  S&T  budgets,  the  infrastructure  to  support  it,  and  the  perceived 
payoff.  Fundamentally,  ways  must  be  found  to  implement  workflow  in  S&T  that  take  a  risk- 
averse  approach  (e.g.  incremental  implementation).  Implementation  costs  must  be  carefully 
controlled,  and  only  those  functions  that  are  truly  value-added  and  that  would  actually  be  used  by 
program  managers  should  be  implemented. 

3.3.2  User  Needs  for  Workflow  Tools 

As  noted  above,  the  interviewees  in  the  later  sessions  benefited  from  the  work  accomplished  in 
the  early  sessions.  The  last  three  sets  of  interviewees  were  provided  Avith  a  set  of  functional 
needs  which  they  were  asked  to  review  and  embellish.  The  workflow  tool  functional  needs  that 
were  provided  to  the  latter  three  interview  groups  were  as  follows: 
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1.  Ad  hoc:  The  tool  should  be  capable  of  implementing  variable  or  infrequently-used 
processes. 

2.  Application  Integration:  The  tool  should  be  able  to  use  a  variety  of  COTS  applications 
together,  seamlessly,  to  support  an  overall  process. 

3.  Audit  Trail:  The  tool  should  have  the  capability  of  maintaining  an  audit  trail  of  work 
status  and  key  decisions.  Most  projects  span  several  years,  during  which  personnel  turnover 
and  loss  of  corporate  memory  can  cause  problems.  Capturing  this  information  can  alleviate 
some  of  these  problems.  Archiving  and  backing  up  the  data  are,  therefore,  necessary. 

4.  Calendar  Support:  The  tool  should  support  task  and  personnel  scheduling,  automatic 
notification  of  events,  and  schedule-conflict  resolution.  The  tool  should  be  network-enabled 
to  support  scheduling  among  distributed  IPT  members. 

5.  CDRL  (Contract  Data  Requirements  List)  Reduction:  The  tool  should  be  capable  of 
interfacing  with  COTS  software  (word  processing,  presentations,  graphics,  and  scheduling) 
in  order  to  allow  contractors  to  create  weekly/monthly  reports  capturing  the  pertinent 
program  information  in  any  format.  The  tool  should  allow  the  generation  and  distribution  of 
test  reports,  procedures,  and  plans,  or  any  typical  CDRL  item  in  electronic  form.  Of  course, 
there  will  always  be  items  where  a  "hard  copy"  is  required. 

6.  Coordination:  The  tool  should  facilitate  setup  and  use  of  data  and  information.  This 
capability  includes  the  dispersion  of  data  and  information  resulting  from  the  execution  of  an 
assignment  or  application.  The  tool  should  provide  collaboration  on  and  synchronization  of 
tasks  and  activities.  When  processes  include  the  routing  and  coordination  of  documents,  the 
tool  should  maintain  the  status,  due  dates,  etc.  for  such  documents. 

7.  Customization  to  Particular  Processes:  The  tool  should  be  customizable  and  tailorable 
either  through  a  built-in  development  environment  and/or  APIs.  This  feature  would  allow 
greater  flexibility  in  implementing  workflow  in  the  S&T  IPPD  Process. 

8.  Document-Centered:  The  tool  should  be  capable  of  implementing  workflow  defined  by 
processing-routed  documents  or  work  packages. 

9.  Group/User-Centered:  The  tool  should  be  capable  of  implementing  the  workflow 
process  of  a  general  set  of  tasks  to  be  performed  by  multiple  users  in  a  coordinated  fashion. 

10.  Large-Scale  Standardization:  The  tool  must  be  standardized  across  a  large 
organization  such  as  Phillips  Lab  to  allow  program  managers  to  acquire  and  turn  over 
programs  easily.  Too  much  personalization  of  the  tool  will  inhibit  this  turnover. 

11.  Match  Government  File  Systems:  Systems  to  electronically  track  records  must  agree 
with  office  systems  already  extant  to  track  hard  copies  of  documents.  Otherwise,  people 
would  be  stuck  with  two  systems  to  learn  and  would  use  both  poorly. 

12.  Open  Systems:  The  tool  should  operate  across  platforms  and  operating  environments. 


45 


13.  Production  Processes:  The  tool  should  have  the  capability  to  implement  repetitive 
processes,  following  codified  rules. 

14.  Progress  Tracking:  The  tool  should  have  the  capability  to  track  the  progress  of  and 
measure  the  effectiveness  of  business  processes.  The  tool  should  be  capable  of  capturing 
process  metrics  for  analysis. 

15.  Project  Management:  The  tool  should  provide  the  capability  to  assign  activities  and 
work  packages,  including  negotiation  of  task  resources,  objectives,  and  schedules.  The  tool 
should  enable  the  manager  to  prioritize  tasks  and  enable  team  members  to  react  to  priorities. 
The  tool  should  prompt  team  members  when  activities  need  to  be  done  or  when  a  suspense  is 
due.  The  tool  should  also  feature  management  reporting. 

16.  Security/Limited  Access:  The  tool  should  handle  some  security  aspects  of  programs. 

If  the  project  includes  proprietary  or  classified  aspects,  it  should  be  able  to  reference  them 
There  should  be  a  procedure  by  which  a  person  can  be  approved  or  refused  access  to  that 
data. 

17.  System  Facilitator:  First,  someone  with  clout  must  be  established  to  enforce  use  of  the 
system.  It  cannot  be  optional.  Second,  the  workflow  tool  should  support  task  facilitation  and 
business-rule  enforcement. 

18.  Visualization:  The  tool  should  provide  a  graphical  capability  to  define  and  test  proposed 
processes. 

The  voting  results  shown  in  table  8  represent  the  core  set  of  features  agreed  upon  by  the  majority 
of  participants  in  each  of  the  interviews.  For  the  purposes  of  this  analysis,  a  “core”  feature  was 
ranked  in  the  top  twelve  to  fifteen  features  in  at  least  two  of  the  interviews.  The  core  features 
identified  for  workflow  tools  were: 

1 .  Application  Integration, 

2.  Audit  Trail, 

3.  Calendar  Support, 

4.  Coordination, 

5.  Customizable, 

6.  Document-Centered, 

7.  Group/User-Centered, 

8.  Open  Systems, 

9.  Production  Processes, 

10.  Progress  Tracking, 

1 1 .  Project  Management, 

12.  Security, 

13.  System  Facilitator,  and 

14.  Visualization. 
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Table  8.  Major  Features  Desired  for  Workflow  Tools 


Interview#! 

5 -6  June  1996 

Wright  Laboratory  ’ 

Interview  #2 

19 -20  June  1996 
Wright  Laboratory 

Interview  #3 

14 -15  August  1996 
Armstrong 
•  Laboratory 

Interview  #4 

18 -19  Sept  1996 
Phillips  Laboratory  ij; 

Interview  #5 

16 -17  October  1996 
Rome  Laboratory  ; : 

Ad-Hoc 

Application  Integration 

Application  Integration 

Application  Integration 

Application  Integration 

Audit  Trail 

Audit  Trail 

Audit  Trail 

Audit  Trail 

Audit  Trail 

Calendar  Support 

Calendar  Support 

Coordination 

Coordination 

Coordination 

Coordination 

Coordination 

Customizable 

Customizable 

Customizable 

Customizable 

Customizable 

Document-Centered 

Document-Centered 

Document-Centered 

Document-Centered 

Document-Centered 

GroupAJ  ser-Centered 

GroupAJ  ser-Centered 

GroupAJser-Centered 

GroupAJser-Centered 

GroupAJser-Centered 

Large  Scale 
Standardization 

Open  Systems 

Open  Systems 

Open  Systems 

Open  Systems 

Open  Systems 

Production  Processes 

Progress  Tracking 

Progress  Tracking 

Progress  Tracking 

Progress  Tracking 

Progress  Tracking 

Project  Management 

Project  Management 

Project  Management 

Project  Management 

Project  Management 

Security 

Security 

Security 

Security 

System  Facilitator 

System  Facilitator 

Visualization 

Visualization 

Visualization 

Visualization 

3.3.3  Market  Assessment  for  Workflow  Tools 

The  market  assessment  for  workflow  was  based  on  an  evaluation  of  available  workflow  tools 
with  respect  to  the  functional  needs  listed  in  paragraph  3.3.2,  as  well  as  an  assessment  of  a  few 
key  imderlying  technical  requirements  needed  to  address  the  user-defined  functions.  Only  tools 
that  addressed  the  following  key  functions  were  further  assessed: 

•  Web  enabled  --  Capable  of  supporting  workflow  functions  across  a  distributed 
environment, 

•  Industry-wide  platform  compatibility  —  Compatible  with  Windows  NT,  and 

•  Commitment  to  support  Common  Object  Oriented  Request  Broker  Architecture 
(CORBA)  standards  for  inter-application  integration. 

Over  130  workflow  tools  were  screened  in  a  preliminary  review,  but  the  more  detailed 
assessment  below  is  limited  to  the  only  three  workflow  tools  that  were  Web  enabled  at  the  time 
of  the  review.  Because  of  the  importance  of  the  Internet  as  an  infrastructure  to  support  platform- 
independent  client-server  applications,  it  is  likely  that  many  more  workflow  tools  will  become 
Web-enabled  in  the  near  future.  A  discussion  of  market  trends  in  terms  of  industry  initiatives 
follows  the  workflow  tool  summary,  and  this  section  of  the  report  concludes  with 
recommendations  for  future  workflow  tool  selection  and  implementation. 


47 


3.3.3.1  Major  Features  of  Interest 

Table  8  reveals  that  some  features  were  important  to  only  one  or  two  of  the  participating  groups, 
reinforcing  the  notion  that  each  program  will  have  different  needs  based  on  the  type  of  research 
and  the  maturity  of  the  technology.  The  features  in  table  8  are  listed  alphabetically.  The  initial 
interview  identified  two  issues  —  ease  of  use  and  affordability  —  that  were  considered  important 
but  are  not  listed  here.  Ease  of  use  is  a  standard  requirement  for  any  tool  to  be  effective. 
Affordability  (in  terms  of  the  cost  of  tool  purchase,  deployment,  and  maintenance)  is  also 
essential.  Desired  features  are  lined  up  across  the  table  so  that  it  is  easy  to  see  the  degree  of 
consensus. 

3.3.3.2  Workflow  Tool  Candidates 

The  market  assessment  for  workflow  tools  began  with  an  initial  screening  of  over  130  tools 
which  purport  to  support  workflow.  This  large  list  was  quickly  culled  to  a  handful  of 
applications,  three  of  which  are  Web-enabled.  A  key  criterion  was  the  ability  of  a  workflow  tool 
to  support  S&T  business  processes  among  geographically  distributed  members  of  an  IPT.  This 
portion  of  the  task  was  somewhat  attenuated  due  to  increased  emphasis  on  the  tool  interviews. 
The  loss  is  small,  however,  because  regardless  of  what  is  published  in  this  report  with  respect  to 
workflow,  it  will  be  out  of  date  within  six  months.  The  market  is  changing  rapidly,  and  there 
will  be  a  major  shakeout  with  a  handful  of  clear  market  winners  dxiring  the  next  two  to  three 
years. 

In  spite  of  the  current  relative  immaturity  of  this  market  and  the  emerging  nature  of  Web-enabled 
applications  in  general,  we  were  able  to  find  three  Web-enabled  workflow  tools  that  appear  to 
meet  many  of  the  user-identified  needs.  The  tools  identified  as  potential  candidates  for  workflow 
are  shown  in  table  9. 

3.3.3.3  Workflow  Tool  Evaluations 

The  workflow  tools  identified  in  table  9  were  evaluated  against  the  major  tool 
features/capabilities  described  above  in  paragraph  3.3.3. 1.  The  results  of  this  evaluation  are 
presented  in  table  10.  Each  tool  was  eveduated  against  the  desired  features  based  on  vendor 
claims  and,  where  possible,  hands-on  testing.  The  key  to  the  table  is: 

1 .  •  indicates  the  tool  offers  strong  support  for  the  feature  or  function. 

2.  >  indicates  the  tool  offers  medium  support  for  the  feature  or  function. 

3.  O  indicates  the  tool  offers  only  limited  or  weak  support  for  the  feature  or  function. 

4.  A  blank  cell  indicates  the  tool  offers  no  support  for  the  feature  or  function. 
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Table  9.  (Web  Enabled)  Workflow  Tool  Candidates 


rust 

-J  Vi  . . . 

4 

s 

IE 

liS 

Action 

Technologies 

Metro 

PC  Based, 
web-enabled 
Workflow 
System 

$4,995  Developer  Tool 
(1  required) 

$199  each  additional  Analyst 
version 

Client  licenses: 

$27.5k/  200  user,  55k/500  user 
$82,500,  unlimited 

$9,995  Metro  1.1  Developer 
Starter  Kit  (includes  30  users) 

1301  Marina  Village 
Pkwy,  Suite  100 

Alameda,  CA  94501- 
1028 

(800)  WORKFLOW 

Universal 

Energy 

Systems 

Track  It 

PC  Based, 
web-enabled 
Workflow 
System 

$18,000  KI  SHELL  Devel.  Ver. 

$1,200  Additional  Runtime 

licenses 

(KI  SHELL) 

$800  per  concurrent  user 

Dr.  Jay  Ramanathan  - 
Dir.  of  Knowledge 

Integ.  Center 

5162  Blazer  Pkwy, 

Dublin,  OH  43017 
(614)  792-9993 

Lotus 

Lotus 

Notes 

$68 1 ,  Notes  4.0  for  Server 
(1  required) 

$407,  Notes  4.0  Client 
(1  required) 

$112,  Notes  4.0  DT  Client  (1  per 
user) 

$900,  10  License  Pack 
$1,760, 20  License  Pack 
$895,  Starter  Pack  (includes  1 
Server,  1  Client  and  2  Desktop 
Clients) 

55  Cambridge  Pkwy 
Cambridge,  MA  02142 
(617)  577-8500 

Quality 

Decision 

Management, 

Inc 

Business 

Builder 

(Requires 
Lotus  Notes) 

$1895  per  Server  ~  Requires 
Notes  Release  4  and  a  Windows 
Client.  Runs  on  all  server  & 
client  platforms  supported  by 
Notes  Release  4 

200  Sutton  Street,  Suite 
225 

North  Andover,  MA 
01845  (508)  688-8266 

It  is  stressed  that  the  tool  evaluation  was  accomplished  by  reviewing  vendor  literature  and  third 
party  reports.  In  some  cases,  demonstration  copies  of  the  tools  were  obtained  and  reviewed.  In 
other  cases,  vendors  demonstrated  the  tools.  The  reader  is  cautioned  that  the  only  way  to  verify 
software  vendor  claims  is  to  actually  employ  the  software  in  a  demanding  application.  The 
resources  on  this  preliminary  analysis  task  were  not  sufficient  to  accomplish  that  level  of  test  and 
evaluation.  It  is  also  stressed  that  in  the  area  of  workflow  the  marketplace  is  extremely  dynamic. 
Workflow  tool  selection  must  involve  the  specifics  of  the  intended  application  and  the  emerging 
developments  in  the  workflow  marketplace. 
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Table  10.  Workflow  Tool  Evaluations 


■r  ll^tURE'J:VALUATEI)'-;l 

Action  Tech.: 
Metro 

UES: 

::tracklt 

Lotus: 

t'Notes 

QDM:  ; 

•Business 
Builder 

Application  IntegKftidn : 

1 

1 

1 

V'-  :■  Audit TraiL'^i\'r\”^  ^ 

1 

;  Coordination 

• 

O 

O 

•  :  Customiz^le  :: 

• 

• 

O 

1 

Document-Centered 

• 

• 

o 

o 

Group/User’ Centered. 

1 

» 

o 

o 

•  ,  Open  Systems 

• 

1 

• 

• 

Production  Prdcesses  .i  .- 

• 

• 

o 

• 

» 

o 

o 

.Project  Management 

• 

1 

o 

o 

Security 

» 

1 

• 

• 

'  System  Facilitator  iv  ' 

» 

1 

o 

; ,  Visualization;  •! 

1 

• 

o 

3.3.4  Market  Trends  in  Workflow  Tools 

As  indicated  above,  the  marketplace  for  workflow  tools  is  extremely  dynamic.  The  fundamental 
issue  will  be  ease  of  implementation,  which  translates  into  development  and  support  costs. 
Because  workflow  is  by  nature  a  complex  endeavor,  reducing  the  overhead  associated  with 
workflow  applications  will  require  the  software  to  be  more  “intelligent.”  The  concept  of 
“intelligent  agents”  is  one  of  the  most  promising  approaches  to  making  workflow  software  more 
powerful,  yet  easier  to  use. 

Within  the  next  five  years,  the  majority  of  workflow  applications  will  be  implemented  in  Internet 
or  Internet  environments.  Microsoft’s  next  generation  object-oriented  operating  system, 
currently  named  Cairo,  will  provide  highly  integrated  Internet  capabilities.  The  application 
suites  and  workflow  tools  that  run  on  Cairo  and  other  operating  systems  will  automatically 
benefit  from  this  integration  so  that  the  notion  of  being  “Web  enabled”  will  be  essentially 
universal. 
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It  is  likely  that  within  three  to  five  years,  a  new  breed  of  Internet  development  tools  based  on 
Java  will  emerge  to  provide  a  more  robust  capability  for  integrating  numerous  applications 
within  a  workflow  context.  A  key  to  distributed  computing  will  be  the  emergence  of  standards 
such  as  the  Common  Object  Request  Broker  Architecture  (CORE A).  They  will  enable  a  new 
level  of  application  integration  and  information  via  a  common  set  of  Application  Program 
Interfaces  (APIs).  Many  of  the  workflow  functions  listed  above  will  ultimately  be  built  into 
commercial  operating  systems  or  will  be  available  as  extensions  to  popular  operating  systems. 
Workflow  systems,  however,  will  always  require  tailoring  to  the  specific  business  applications  or 
environments  in  which  they  are  deployed.  The  proliferation  of  workflow  tools  will,  therefore, 
reinforce  the  current  trend  toward  process-driven  metrics  and  management. 

3.3.5  Recommendations  for  Future  Development 

The  current  assessment  of  workflow  tools  revealed  the  following  fundamental  shortfalls  in 
today’s  applications: 

•  Web  (Internet)  enabled  capabilities  are  not  sufficiently  rich. 

•  Tools  are  immature  and  are  often  limited  by  the  platforms  they  support. 

•  The  tools  are  too  difficult  to  use  and  require  a  major  commitment  to  implement  and 
support. 

•  Security  issues  at  all  levels,  including  access  and  encryption,  have  not  been 
adequately  addressed. 

•  Issues  between  government  and  contractor  entities,  as  well  as  issues  between 
contractors,  remain  to  be  resolved.  They  are  particularly  poignant  in  a  workflow 
environment  where  intersections  must  be  established  between  different  business 
processes. 

•  Process  models  for  the  business  side  of  the  enterprise  that  are  analogous  to  those  for 
the  product  side  of  the  enterprise  do  not  generally  exist.  These  models  are  required 
for  effective  workflow  implementation. 

Certain  issues,  such  as  Web  enabled  capabilities  and  product  maturity,  are  being  addressed  by 
software  vendors.  Implementation  issues  are  also  being  addressed  by  commercial  tool  vendors. 
Tools  will  become  easier  and  more  cost-effective  to  implement,  and  security  capabilities  are  a 
design  goal  for  most  Web-enabled  products  today.  However,  the  world  of  commercial  software 
tools  and  vendors  is  not  addressing  the  prime-supplier  value-chain  issues  that  require  workflow 
to  be  employed  across  multiple  organizations,  including  those  of  both  government  and  industry, 
particularly  in  the  context  of  defense  research  and  acquisition. 

Because  commercial  software  vendors  are  investing  heavily  in  the  development  of  improved 
workflow  tools,  the  Air  Force  should  closely  follow  commercial  developments  in  this  area,  and 
invest,  instead,  in  workflow  applications  -  the  tailoring  of  commercial  workflow  tools  to  support 
defense-unique  needs.  Because  workflow  touches  almost  every  aspect  of  a  process,  workflow 
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applications  would  likely  be  complex  and  expensive  to  implement.  (This  is  less  true  of  group- 
consensus  applications  because  they  can  be  targeted  at  specific  problems  and,  hence,  can  be  of 
more  limited  scope.)  With  respect  to  the  S&T  IPPD  initiative,  the  extent  the  workflow 
applications  provide  return  on  the  investment  required  for  their  implementation  remains  to  be 
seen.  We  do  not  anticipate  that  workflow  will  play  a  significant  role  in  FY97  S&T  IPPD  pilot 
programs.  There  are,  however,  applications  for  workflow  that  should  be  considered  which  may 
impact  S&T  programs  in  the  future. 

One  of  the  most  important  defense  needs  is  to  address  the  problem  of  enterprise  integration 
where  the  “enterprise”  consists  of  one  or  more  prime  contractors  and  numerous  suppliers  (even 
multi-tier  suppliers).  This  situation  would  be  extremely  rare  for  S&T  but  is  common  during 
acquisition.  Commercial  workflow  solutions  have  not  really  addressed  the  issue  of  workflow 
across  the  extended  enterprise  described  here.  We,  therefore,  recommend  two  S&T  efforts  to 
explore  the  application  of  workflow  to  that  environment.  The  first  recommendation  addresses 
the  need  for  improved  methods  to  model  the  enterprise  processes  that  the  workflow  system 
would  support.  The  second  recommendation,  building  on  the  first,  outlines  an  experiment  in 
applying  workflow  to  the  enterprise  processes  in  a  defense  context. 

3.3.5.1  Recommendation  1:  Heuristic  Models  for  Process  Representation  (HMPR) 

Background:  Companies  are  developing  sophisticated  Process  Capability  Models  (PCMs)  which 
link  their  manufacturing  process  capabilities  to  design  features.  PCMs  improve  the  design 
process  by  enabling  products  to  be  “manufactured  in  a  computer”  before  being  submitted  to  “real 
world”  manufacturing.  These  PCMs  are  not  manufacturing  simulations,  but  represent  a  next- 
generation  set  of  heuristic  “design  rules”  with  which  designers  interact  in  “real  time”  as  they 
make  critical  design  decisions.  Companies  are  succeeding  in  building  these  heuristic 
representations  of  the  manufacturing  process  at  far  less  expense  (and  complexity)  than  is  the  case 
with  more  conventional  modeling  and  simulation.  The  question  is  whether  analogous  PCMs 
could  be  built  for  the  business  processes  that  drive  an  organization.  If  so,  they  could 
dramatically  improve  our  capability  to  understand  and  continuously  improve  those  processes. 

Relevance  to  Workflow:  Workflow  systems  implement  and  help  enforce  business  processes. 
Ultimately,  a  workflow  system  should  support  continuous  process  improvement.  That  kind  of 
support  is  not  possible,  however,  without  well-defined  ways  to  represent  the  processes  which  the 
workflow  system  is  supporting. 

Purpose;  The  purpose  of  this  effort  would  be  to  demonstrate  the  viability  of  a  heuristic  business- 
process  representation  that  would  be  analogous  to  a  current  heuristic  manufacturing-process 
representation.  The  approach  should  support  the  representation  of  business  processes,  including 
the  Air  Force  R&D  process,  in  commercial,  network-enabled  workflow  applications.  The 
objective  is  to  substantially  reduce  the  cost  and  complexity  of  implementing  improved  business 
process  support,  control,  and  improvement  strategies  across  enterprises  of  any  scale. 

Level  of  Effort  and  Timing;  The  recommended  timing  and  level  of  effort  (in  person-hours)  for 
this  program  are  as  follows. 
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FY97  FY98  FY99  FYOO 

Person-Hours  350  6500  6500  4000 

Deliverables  Plan/Strategy  Draft  Semantic  Semantic  Demonstration 


3.3.5.2  Recommendation  2:  Enterprise  Integrated  Workflow  (EIW). 

Background:  A  key  defense  concern  today  is  to  maintain  an  adequate  defense  manufacturing 
capability  in  the  face  of  declining  defense  budgets  and  low-volume,  time-delayed  acquisition 
schedules.  A  number  of  strategies  are  emerging  to  attempt  to  address  this  problem.  The  Lean 
Aircraft  Initiative  (LAI)  and  its  extension  to  the  supplier  base,  the  Lean  Supplier  Initiative  (LSI), 
are  two  of  the  more  promising  approaches  to  ensuring  a  defense  industrial  base  sufficient  to  meet 
a  wide  range  of  potential  threats.  LAI  and  LSI  are  exploring  the  application  of  the  principles  of 
Lean  Manufacturing,  which  have  been  successful  in  the  commercial  domain,  to  the  defense 
manufacturing  environment;  in  particular,  to  airframe  manufacturing.  The  focus  of  much  of  this 
activity  is  on  business  process  improvement.  The  tools  to  support  these  new  business  processes 
in  the  defense  domain  are  immature,  and  workflow  tools  that  are  being  developed  to  support 
commercial  enterprises  are  often  not  appropriate. 


Purpose:  Explore,  demonstrate,  and  measure  the  benefits  of  using  network-enabled  workflow 
tools  to  support  enterprise  integration  throughout  the  defense  prime-supplier  value  chain. 
Determine  the  required  functionality  of  workflow  tools  to  support  business  process  improvement 
in  that  environment,  the  appropriate  metrics  to  measure  that  improvement,  and  the  special  needs 
imposed  by  the  nature  of  defense  manufacturing. 


Level  of  Effort  and  Timing:  The  recommended  timing  and  level  of  effort  (in  person-hours)  for 
this  program  are  as  follows. 


Person-Hours 

Deliverables 


FY98 
2000 
Strategy  & 
Metrics 


FY99 

10000 

Progress  Repts 
&BPR 
Functional 
Specification 


FYOO 

10000 

Preliminary 

Demonstration 


FY01 

8000 

Complete 

Functional 

Specification 


FY02 

5000 

Demonstration  & 
Final  Technical 
Report 


3.4  (Design)Value Analysis 

One  of  the  hurdles  in  applying  IPPD  to  new  technology  development  is  quantifying  impact  of 
critical  design  or  architecture  decisions  on  transition  cost  and  risk.  The  objective  of  value 
analysis  is  to  enable  the  quantification  of  the  cost,  risk,  and  relative  value  of  competing 
technologies  (or  design/architecture  alternatives).  In  the  case  of  S&T,  it  also  facilitates 
technology  investment  and  design  decisions  earlier  in  the  technology  development  process. 

The  S&T  IPPD  value  scorecard  shown  in  figure  2enables  the  comparison  of  performance 
capability,  part/supplier  capability,  process  capability,  supportability,  life  cycle  cost,  and  risk  in 
an  integrated  matrix.  As  figure  2suggests,  the  S&T  IPPD  value  scorecard  combines  principles 
from  industry  (Six  Sigma,  Motorola,  and  the  Texas  Instruments  Six  Sigma  scorecard)  and 
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academia  (Prof.  R.  Kaplan,  Harvard,  the  Balanced  scorecard).  The  objective  of  the  S&T  IPPD 
value  scorecard  is  to  enable  the  credible  estimation  of  the  relative  value,  costs,  and  risks  of  new 
technologies  during  their  development. 
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Figure  2.  The  S&T  IPPD  Value  Scorecard 


The  difficulty  in  understanding  the  needs  for  tools  and  methods  to  support  value  analysis  is  that 
this  type  of  analysis  involves  many  disciplines  and  many  types  of  tools.  Additionally,  the 
methodology  for  value  analysis  is  still  under  development.  The  constellation  of  tools  required 
depends  on  the  nature  of  the  project.  In  order  to  help  focus  the  interview  discussions,  a  scenario 
was  presented  which  showed  how  the  S&T  IPPD  value  scorecard  could  be  populated  from 
underlying  technology  (or  design)  worksheets,  as  well  as  how  the  scorecard  could  support 
technology  investment  decisions.  By  facilitating  the  application  of  a  structured  methodology, 
value  analysis  tools  aid  in  tailoring,  then  populating  the  design  worksheets  and  the  scorecard. 

3.4.1  Interview  Results 

The  interviews  for  value  analysis  were  handled  differently  from  those  for  the  other  three  tool 
areas.  As  with  the  other  tool  areas,  participants  responded  to  six  warm-up  questions  following  the 
presentation  of  the  scenario.  However,  rather  than  rank  and  prioritize  tools  features  and 
functions,  the  participants  were  then  asked  to  respond  to  two  focused  questions: 

1.  Which  “pieces  ”  of  these  IPPD  process  activities  (3&4)  might  apply  to  your  program(s)  ? 
(“Tailor”  as  required  and  describe  the  key  functions  associated  with  those  “pieces.  ”) 

2.  What  are  the  tools  and  methods  needed  to  support  those  “pieces”  or  activities? 

Note:  Activity  3  involves  developing  technology  alternatives.  Activity  4  involves  performing  the 
value  analysis. 

Rationale:  the  value  analysis  tool  area  is  sufficiently  broad  that  the  participants  could  not 
address  specific  tool  needs  apart  from  the  context  of  a  specific  program  or  application.  A 
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representative  selection  of  the  responses  to  these  questions  is  provided  below,  followed  by  an 
analysis  of  the  common  themes  that  emerged  from  them. 

What  lower  level  analyses  are  required  to  generate  scorecard  data? 

•  Analyses  for  this  program  include  all  the  “ilities”  (reliability,  maintainability, 
deployability,  supportability,  etc.)  as  well  as  basic  performance  and  cost  issues.  The 
data  . . .  include  ballpark  threshold  and  goal  metrics  for  each  aspect  of  each  design. 

•  I  anticipate  making  very  top-level  engineering  estimates.  I  do  not  anticipate  running 
simulations  or  models  in  the  near  term. 

•  Those  yielding  excellent  knowledge  of  the  concept  and  its  encompassing 
technologies. 

•  For  selection  systems,  the  capability  to  identify  quality  personnel  is  analogous  to 
system  performance.  We  collect  empirical  data  and  conduct  statistical  analyses  on 
selection  systems  -  ones  in  use  and  possible  alternatives  -  to  evaluate  differences  in 
performance  capabilities. 

•  The  different  designs  come  out  of  people’s  knowledge  of  what’s  available  that  could 
be  used  for  the  purjpose.  The  rest  of  the  data  would  likely  be  subjective.  We’ve 
discussed  how  this  process  is  done  subjectively  and  unsystematically  in  developing  a 
tutor,  so  in  some  sense  these  factors  are  subjectively  weighed  now.  I  suppose  there 
might  be  some  objective  information  available  some  of  the  time  for  filling  in  some  of 
the  columns,  but  I  think  this  would  be  serendipity  and  you  couldn’t  count  on  it. 

What  tools  do  you  use  to  assess  the  “ilities”? 

•  For  this  program,  some  of  the  tools  will  be  built  into  the  architecture  development 
tools  (such  as  design  rules).  In  other  cases,  the  “tools”  will  be  a  review  panel  of 
experts. 

•  We  don’t  really  use  tools  for  this. 

•  I’m  not  aware  of  any  tools  to  assess  the  “ilities.” 

•  There  are  no  tools  that  I  am  familiar  with  to  assess  the  “ilities.”  Perhaps  the 
acquisition  community  should  take  on  the  task  of  providing  the  labs  with  this  data? 

•  A  bunch  of  databases  about  the  frequency  of  field-service  failures  and  the  causes  for 
removal-from-service  give  us  our  best  handle  on  Reliability  and  Maintainability 
(R&M).  Vendor  data  for  rejection  rate  in  manufacturing  gives  us  our  best  shot  at 
quantifying  producibility. 

•  In  the  current  environment,  we  use  none  (at  least  no  software  tools).  The  process  and 
procedures  to  assess  reliability,  maintainability,  skill  levels,  and 
Service/Maintain/Replace  (SMR)  are  well  known.  There  are  just  few  formal  tools  in 
use  (maybe  due  to  lack  of  availability). 
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•  One  tool  I  use  to  address  reliability  issues  is  MIL  HDBK  217,  Reliability  of 
Electronic  Parts  and  Systems.  There  are  others. 

•  Our  tools  include  system-level  design  tools  to  define,  analyze,  and  allocate 
requirements  from  system  to  subsystem  level.  Then,  as  these  subsystems  are  actually 
designed,  more  realistic  data  is  rolled  up  to  ensure  the  system-level  requirements  are 
being  met.  We  use  the  traditional  hardware  design  tools  available  commercially 
today. 

•  We  usually  assess  the  “ilities”  qualitatively  through  past  experience  and  an 
understanding  (educated  guess?)  of  what  the  issues/drivers  will  be. 

•  I  agree  with  the  above  comment.  It  is  all  guess-work  based  on  the  person’s  vision  of 
the  end  item  and  the  state  of  the  world  in  that  area  of  expertise. 

How  do  you  estimate  and  document  risks?  (List  tools.) 

•  I  don’t  think  we  do  this  in  many  of  our  programs. 

•  It  is  largely  based  on  experience  and  technical  training.  It  is  largely  subjective. 
Perhaps  more  time  should  be  spent  in  this  activity. 

•  Risks  . . .  can  be  estimated  by  running  simulations  of  proposed  technologies  versus 
real-world  test  systems.  The  results  will  show  how  well  the  technology  meets  the 
project  goals.  The  risk  should  surface  in  the  outlying  cases  where  the  goals  were  not 
met. 

•  Technological  risks  are  identified  by  a  group  of  experts. 

•  Risk  estimates  are  typically  based  on  historical  experience  with  past  projects.  In  my 
experience,  the  documentation  of  program  risks  has  been  sparse  at  this  lab.  I  have 
seen  and  worked  on  other  programs  that  have  detailed  risk-management  plans.  They 
address  risks,  mitigation  efforts,  impacts  on  schedule,  etc. 

•  Informally.  The  only  formal  documentation  is  in  the  Notes-To-The-Buyer  section  of 
a  contract  award.  Our  risk  estimates  are  “WAGS”  based  on  gross  historical  measures. 
(E.g.,  has  such  &  such  functionality  been  demonstrated?  If  so,  how  stable  was  it?) 

How  do  you  address  process  capability  (manufacturing,  software  development,  training 
development,  etc.)  in  technology  development? 

•  Right  now,  we  address  process  capability  by  expert  judgment  (i.e.,  subjectively)  and 
roll  that  assessment  into  our  judgments  on  source  selection,  contract  options,  etc. 
There  are  sufficiently  few  sources  for  each  technology  that  a  finely  tuned  assessment 
is  seldom  necessary. 

•  For  development  of  software  for  selection  systems  developed  imder  6.2  and  6.3 
programs,  process  capability  is  not  strictly  applicable. 
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•  In  the  S&T  world  it  is  extremely  difficult.  However,  there  are  a  lot  of  tools  used  to 
address  process  when  dealing  with  EMD/Deployment  programs.  S&T  tries  to  push 
the  state-of-the-art,  and,  therefore,  introduces  more  risk  Sometimes  these  programs 
highlight  where  processes  are  insufficient  or  need  more  refinement. 

•  Informally,  not  systematically. 

•  From  the  past  experience  of  contractors  and  from  previous  programs. 

•  Process  capability  is  very  hard  to  quantify,  very  subjective.  There  are  a  few  metrics 
that  are  appropriate  for  each  category,  but  they  are  not  well  known  or  frequently  used. 

•  6.3  programs  address  scale-up  and  repeatability.  Often  we  depend  on  a 
manufacturing  science  or  manufacturing  technology  effort  to  improve  yield  and 
manufacturability. 

•  In  my  experience  with  software  systems,  processing  the  capability  assessment  is  still 
just  an  “art”  -  especially  in  small  and  some  medium-size  companies.  The  larger 
companies  are  still  trying  to  get  a  handle  on  their  processes  program  by  program.  (1 
don’t  believe  the  Capability  Maturity  Model  hype  &  bravado.)  As  for  training 
development,  it’s  not  just  an  art,  it’s  a  “black”  art. 

What  might  keep  you  from  collecting  data  to  populate  these  scorecards? 

•  You  need  readily  available  sources  for  this  data.  If  the  data  are  too  hard  to  collect, 
then  they  will  not  be  used.  It  seems  the  scorecard  approach  is  only  as  good  as  the 
input  data. 

•  Cost  is  an  issue.  How  much  does  it  take  to  complete  the  full  analysis? 

•  Nothing  in  theory,  as  long  as  you  make  a  point  of  trying  to  get  it.  Some  information 
might  be  more  problematical  or  subjective  than  other  information. 

•  It  is  hard  to  envision  how  we  could  use  these  trade-off  tools  when  we  can’t  envision 
what  the  trade-offs  could  be  ahead  of  time;  i.e.,  we  can’t  specify  the  tradeoffs, 
contractually.  Also,  it  is  natural  that  the  contractor  will  select  the  design  that  he 
thinks  is  the  easiest  or  cheapest  to  implement.  The  bottom  line  is  that  usually  only 
the  contractor  has  the  data  to  populate  the  scorecards;  we  don’t. 

•  I  don’t  see  a  problem  in  getting  this  information.  In  my  program  when  beginning 
development  of  the  prototype,  the  contractor  went  through  trade-off  analyses  to  select 
the  best  design  approach.  Yes,  the  contractor  has  the  information  to  do  the 
scorecards,  and  we  have  every  right  to  access  it.  Also,  you  are  in  control  to  select  the 
best  method  for  implementation  ...  the  more  systematic  a  way  to  do  this  as  manager, 
the  better. 

•  I  think  you  can  always  make  an  educated  guess  about  each  data  element.  But,  issues 
such  as  proprietary  technologies  may  keep  you  from  entering  the  best  data  for  a  good 
analysis  of  the  proposed  design. 
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•  In  some  6.3  and  certainly  in  the  6.2  programs,  ignorance  is  the  biggest  barrier. 

Usually  the  focus  of  the  6.2  programs  is  to  collect  this  data  on  a  specific  technological 
approach.  This  means  the  analysis  tools  would  be  most  accurate  at  the  end  of  6.2 
programs  -  if  each  program  produced  the  data  in  a  consistent  format. 

•  The  historical  ways  of  doing  business  for  many  of  our  vendors,  suppliers,  and 
subcontractors  have  never  included  methods  which  produce  the  data  needed  to 
populate  these  scorecards.  The  trick  is  going  to  be  ramping  up  each  of  these 
organizations  to  the  point  where  they  will  produce  the  required  scorecard  data. 

•  The  whole  idea  is  wonderful  as  a  concept.  However,  many  concepts  can't  go  beyond 
this  stage  due  to  political  and  other  real-world  requirements.  I  agree  Avith  the 
comments  earlier  in  the  day  that  said  this  should  be  done  early:  either  prior  to  the  6.3 
contract  award,  or  as  part  of  a  phased  or  delivery-order  approach  to  the  6.3  effort. 
However,  the  money  and  time  required  to  gather  the  data  and  perform  this  analysis 
may  be  difficult  to  find.  Assuming  sufficient  people,  money,  and  time,  I  can't  see  a 
technical  reason  why  this  couldn’t  be  done  on  any  program  if  the  project  leader  is 
given  the  required  freedom. 

Did  the  demonstration  enable  you  to  understand  value  analysis  and  how  it  could  be 
implemented  in  the  S&TJPPD process? 

•  Well,  it  started  to.  I  understand  the  concepts  as  they  apply  to  manufacturing  and  I  can 
think  of  several  ways  of  estimating  (actually  “SWAGing”)  the  numbers  for  our  stuff. 
Again,  we  hope  our  decisions  are  based  on  something  better  than  throwing  a  dart.  If 
so,  we  can  find  ways  to  put  it  in  a  scorecard. 

•  The  demonstration  was  very  instructive.  However,  generalization  to  S&T  areas  other 
than  hardware  and  software  will  take  time  “to  catch  on." 

•  Yes,  the  demonstration  did  describe  how  I  could  use  value  analysis.  The  question 
remains  regarding  where  I  draw  the  information.  My  own  answer  to  this  is  personal 
knowledge.  I  would  get  a  better  feel  for  the  method  by  performing  an  evaluation. 

•  Based  upon  this  demonstration,  I  understand  the  applicability  of  the  value  scorecard 
and  how  one  determines  the  index  of  technical  risk.  I  think  they  would  definitely  be 
beneficial.  At  a  high  level  you  could  identify  the  tradeoffs  between  the  different 
parameters.  The  lower  level  data  could  be  captured  in  other  places  (or  even  linked). 

•  I  got  some  understanding  of  the  process.  But,  I  haven't  the  faintest  idea  how  we  are 
going  to  come  up  v^th  some  of  the  estimates  in  the  scorecard,  especially  for  the 
important  issues  of  producibility  and  reliability’ 

•  I've  got  at  least  a  superficial  vmderstanding,  but  I  fear  the  process  is  subject  to  the 
usual  weakness  of  numerical  evaluation  matrices:  the  process  of  generating  the 
entries  (in  this  case,  for  the  "defect  rates")  is  frequently  subjective,  despite  the 
appearance  of  objectivity. 
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3.4.2  User  Needs  for  Value-Analysis  Tools 


The  tool  needs  were  collected  via  responses  to  two  questions.  The  questions  and  their  responses 
follow. 

Question  1:  Which  **pieces”  of  these  IPPD  process  activities  (3&4)  might  apply  to  your 
program(s)?  ("Tailor”  as  required  and  describe  the  key  functions  associated  with  those 
"pieces.  ”) 

•  I  think  that  both  pieces  would  fit  my  programs.  I  do  both  of  these  steps  now. 

•  I  think  all  the  pieces  could  be  applied.  I  just  think  it  would  take  some  effort  and 
fumbling  to  figure  out  how.  A  lot  might  be  subject-matter-expert  “SWAGs”  and  a  lot 
might  be  just  plain  wrong,  but  it's  worth  trying.  We  could  figure  it  out  pretty  well, 
and  be  better  off  than  we  are  now. 

•  Both  pieces  would  help  me  with  project  continuity  --  to  explain  past  decisions  to  new 
representatives  of  my  customers  and  of  my  chain  of  command.  If  the  data  were 
available  on-line  to  my  customers  and  chain  of  command,  I  might  be  able  to  spend 
less  time  explaining  decisions  and  describing  issues.  Both  pieces  would  help  ensure 
smoother  and  more  cost  efficient  transition  of  my  products  firom  6.3  to  6.4,  then  to  the 
user. 

•  Both  pieces  could  apply  as  presented  in  the  briefings.  We  presently  do  them 
differently  than  briefed,  relying  more  on  softer  approaches  than  on  statistical 
analyses,  etc. 

•  I  believe  that  many  of  the  pieces  are  applicable.  I  like  the  concept  of  including 
factors  pertaining  to  performance,  producibility,  and  cost  in  the  decision.  I  like  the 
way  the  methodology  leads  you  through  these  data  collection  and  analysis  efforts. 

I'm  a  little  concerned  about  the  “goodness”  of  the  data  I  will  put  into  the  process. 
From  my  experience,  I  would  tend  to  use  the  “develop  technology  alternatives” 
activity  immediately.  The  “perform  value  analysis”  activity  would  take  more  effort  to 
leam  to  apply  correctly. 

•  My  view  of  IPPD  is  as  a  tool.  From  my  experiences  and  discussions  with  different 
contractors,  technology  alternatives  are  always  addressed.  The  design  worksheet  is 
just  one  tool  that  can  capture  the  essence  of  why  one  design  “passed  muster,”  while 
others  did  not.  It  adds  an  additional  level  of  traceability  and  corporate  knowledge. 

•  It  will  be  difficult  to  manage  the  many  different  concepts  and  potential  technologies 
that  could  fit  into  the  overall  program.  There  could  be  two  or  three  technologies  that 
support  each  aspect  of  an  architecture  concept  and  each  needs  to  be  considered.  This 
aspect  of  the  program  will  make  it  more  complicated  to  come  up  with  the  worksheet 
cards  to  be  used  in  activities  3  &  4. 
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•  Development  of  the  infonnation  that  becomes  the  columns  and  rows  in  the 
worksheets  is  an  intrinsic  part  of  the  program’s  phase  I.  In  general,  no  serious 
tailoring  should  be  needed.  For  the  scoring  process,  the  scorecard  approach  might  be 
difficult  to  implement  completely  since  some  of  the  uncertainties  might  be  very  large 
and  some  of  the  negative  interactions  might  not  be  folly  characterized.  Scoring  is  part 
of  the  downselect  process,  but  “a  “final  grade”  might  be  more  appropriate  for  the 
program’s  phase  II. 

•  All  of  these  activities  will  be  applied  in  this  program.  The  specific  methodologies, 
tools,  and  processes  you  presented  offer  several  alternatives  for  performing  the 
required  analyses.  Ultimately,  it  will  come  down  to  human  judgment;  but  any  tools 
and  structured  processes  would  help  to  make  the  overall  effort  more  beneficial  and 
would  aid  our  ability  to  reconstruct  the  data.  (I.e.,  another  group  conducting  a  similar 
analysis  will  come  up  with  similar  results.)  In  the  long  run,  being  able  to  reproduce 
the  results  and  understand  the  “logic  measures”  that  went  into  a  study  would  be  the 
best  way  to  ensure  acceptance  of  the  study  results.  The  process  would  also  allow  for 
re-use  of  information  that  is  appropriately  maintained,  and  for  easier  adaptation  of  the 
results  to  changing  needs  and  changing  technologies. 

•  Right  now  I  am  planning  research  concerning  development  of  a  personnel  test  to  help 
select  operators  for  unmanned  aerial  vehicles.  I  am  considering  three  alternatives  in 
my  rough  design:  a  printed  selection  test,  a  computer-administered  selection  test,  and 
a  computer-administered  job-sample  test.  Performance  (predictive  validity, 
reliability  in  the  psychometric  sense),  cost,  and  maintainability  could  be  indexed  for 
these  alternatives. 

Question  2:  What  are  the  tools  and  methods  needed  to  support  those  "pieces”  or  activities? 

•  The  HOQ  is  an  excellent  tool  with  which  to  start  defining  technology  options  and 
with  which  to  start  determining  customer  needs,  but  the  customer-needs  dimension 
will  require  relatively  more  work.  Design  worksheets  and  value  analysis  scorecards 
would  also  be  useful  in  the  process  of  determining  customer  needs.  They  could  be 
used  to  show  the  customers  design  properties  from  which  to  choose,  rather  than  to  ask 
the  customers  to  conjure  up  details  about  their  needs. 

•  The  pieces  could  be  knowledge  of  the  technologies  being  applied  as  well  as 
knowledge  of  their  application.  Using  a  matrix  checklist  of  technology  versus 
application  narrowed  down  the  choices. 

•  A  foil  range  of  tools  could  be  applied  to  various  aspects  of  the  process.  Part  of  the 
value  added  is  to  determine  what  current  tools  exist  and  how  they  could  be  applied. 
However,  the  focus  of  this  project  (ISCP)  is  toward  future  technology  results. 

Having  the  S&T  IPPD  members  as  partners  in  the  process  will  add  the  value  of 
addressing  process  issues  on  S&T  programs  while  improving  the  quality  and 
reproducibility  of  the  data  obtained. 
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•  There  will  be  a  lot  of  engineering  trade  studies  performed  to  select  the  COTS 
hardware  and  software.  As  mentioned  earlier,  there  are  lots  of  available  software 
packages.  Selecting  those  packages  that  meet  the  program’s  functional  requirements 
will  be  a  challenge. 

•  I  want  easy-to-use  tools,  not  ones  that  are  too  complex.  I'm  looking  for  tools  that 
minimize  manual  data  movement  and  automatically  fill  in  the  data  where  needed. 
Spreadsheet-based  tools  are  the  way  to  go  initially.  The  tools  should  use  the  data  you 
have  at  the  time  and  allow  you  to  proceed  with  an  assessment.  Then,  as  you  get  more 
data,  the  tools  should  allow  you  to  enter  them  and  see  their  impact. 

•  I  am  not  sure  what  tools  and  methods  are  needed  yet.  I  suppose  standard  software 
tools  would  be  applicable  to  the  testing  software.  But  our  greatest  difficulty 
concerning  new  systems  is  the  cost/benefit  analysis.  “Best  value”  might  not  be 
sufficient. 

•  We  need  something  simple  to  minimize  its  cost  and  to  show  the  laboratory  it  benefits 
programs  in  the  long  run.  It  must  maximize  the  use  of  existing  COTS  hardware  and 
software  (unless  some  other  organization  can  provide  $  to  help),  provide  security 
(such  as  a  point-to-point  system  with  a  dedicated  server  --  the  only  site  bn  a  Web 
browser),  and  be  easily  managed  by  various  participants.  My  opinion  is  that  we 
should  capitalize  on  Microsoft  products  and  Web  browsers  available  on  the  Internet. 

•  The  survey  process  of  tracing  the  macro  needs  described  in  the  Mission  Area  Plans 
(MAPs)  to  the  detailed  technical  requirements  that  supports  them  is  the  tool  that  will 
be  used  for  supporting  both  activities.  Part  of  that  survey  process  is  not  only  to 
identify  technical  requirements,  but  also  to  address  the  range  of  acceptable 
performance  levels  and  the  tools  that  will  be  most  useful  to  ascertain  those 
performance  levels. 

•  The  first  tool  we  will  need  is  a  groupware  type  tool  to  collect  the  suggested 
architectures,  allow  the  group  to  comment  or  change  the  suggestions,  and  examine  the 
architectures  in  detail.  The  next  tool  we  will  need  is  a  scoring  tool  to  perform  a 
preliminary  analyses  of  all  the  suggested  architectures.  It  should  take  scores  from 
each  team  member  as  well  as  comments  regarding  scoring  rationale.  Then,  it  should 
combine  the  scores  and  output  the  top  three  architectures  along  with  the  rationale  for 
their  selection.  We  will  also  need  a  modeling  tool  for  design  alternatives.  It  will 
have  to  analyze  the  performance  metrics  and  give  data  about  potential  pitfalls  of  the 
design.  It  will  also  have  to  give  first  estimates  for  key  engineering  parameters  and 
include  technological  constraints.  The  final  tool  we  will  need  is  a  document 
generation  tool.  This  tool  would  allow  us  to  collectively  prepare  the  two  reports  in 
the  study.  It  must  provide  configuration  management  and  restricted  access  for 
proprietary  information. 

•  I  think  guidelines  should  be  developed  to  itemize  as  many  factors  as  possible  that 
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contribute  to  risk,  and  various  algorithms  should  be  proposed  for  arriving  at  a  single 
risk  factor.  I  think  this  v^ould  be  a  very  complicated  exercise,  but  it  would  provide 
valuable  food  for  thought  for  those  evaluating  risk. 


3.4.2. 1  Value  Analysis  Participant  Feedback:  General  Observations 

The  following  general  observations,  concerning  value  analysis  and  value  scorecard  tools,  are 
based  on  the  above  comments  plus  informal  feedback  from  the  participants. 

1 .  Flexibility,  that  is,  the  support  that  these  tools  provide  to  enable  users  to  tailor  them  to 
specific  project  needs,  is  a  critical  requirement  for  value  analysis  tools. 

2.  The  extent  to  which  various  “ilities”  are  considered  during  S&T  varies  widely  and  is 
dependent  on  the  nature  of  the  technology.  It  is  much  harder  to  understand 
“producibility”  and  even  “reliability”  in  some  contexts  (e.g.,  an  architecture  specification, 
reference  model,  or  training  model)  than  in  others. 

3.  The  importance  of  various  “ilities”  varies  widely  and  is  technology-dependent.  For 
example,  long-term  reliability  is  paramoimt  for  space-based  applications  and  may  not  be 
an  issue  at  all  for  disposable  applications.  It  may  have  no  meaning  for  certain  areas  such 
as  a  standards  specification. 

4.  The  notion  of  “quality”  in  S&T,  where  the  result  might  be  an  advanced  technology 
demonstration,  is  very  different  than  in  an  industrial,  medium-to-high- volume-production 
environment.  It  can,  therefore,  be  difficult  to  think  in  terms  of  “defects”  in  S&T, 
although  in  many  cases  it  is  effective  to  do  so. 

While  it  has  been  applied  successfully  in  industrial  contexts  that  are  similar  to  6.3  S&T,  the 
concept  of  value  analysis  in  S&T  is  new  and  not  well  defined.  It  must  be  employed  in 
actual  case  studies  through  the  S&T  pilots  in  order  to  achieve  more  maturity  and  to 
provide  a  better  understanding  of  how  it  might  reduce  transition  cost  and  risk. 


$.4.2.2  Major  Features  of  Interest 

Six  key  user  needs  are  documented  in  table  1 1 .  Some  of  these  needs,  such  as  Ease  of  Use,  are 
standard  requirements  for  all  software  tools.  However,  the  table  suggests  some  ways  to  measure 
the  extent  to  which  those  needs  have  been  addressed  in  specific  implementations. 
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Table  11.  User-Needs  Summary  for  Value  Analysis  Tools 


'  ’'Illiqiiirenieiit}'  ':.v  ;* 

,  ''.Bescription/Featares 

.  How  Measured 

1 

Easy  to  Use 

The  overall  “look  and  feel”  of  the  interface  should 
be  highly  intuitive,  follow  the  Microsoft  User 

Interface  Guidelines  for  Windows  applications, 
contain  a  complete  help  system,  contain  set-up 
wizards,  and  eliminate  the  need  for  a  user  manual 
except  for  occasional  references.  It  must  be  self- 
consistent  throughout  all  screens  and  modules. 

•  #  Hours  of  training  required  for 
effective  use  by  computer- 
literate  personnel. 

•  User  feedback  and  response 
ratings. 

•  #  Hours  required  for  mid-level 
program  managers  to  set  up  and 
modify  scorecards  and 
worksheets. 

2 

“Anytime,  Anywhere” 

Application  modules  should  support  collaboration 
among  geographically  distributed  IPT  members  in 
the  development  of  scorecards,  worksheets;  and  in 
reaching  consensus  on  scorecard  values  where 
historical,  test,  or  experimental  data  is  lacking. 

•  User  feedback  response  ratings 
on  how  easily  and  effectively 
the  PATA  components  can  be 
used  by  multiple  IPT  members 
over  the  Internet. 

•  %  of  functions  that  can  be 
accomplished  over  a  network. 

3 

Open  Systems  on 
Popular  Platforms 

The  software  should  be  compatible  with  open 
systems  standards  and  should  run  on  popular 
platforms. 

Integration  should  be  via  industiy  standard  and 
popular  protocols  including  OLE  and  MAPI 

Clients  should  be  platform-independent  since 
applications  are  accessed  via  a  Web  browser 
interface.  Applications  should  support  all  major 
browsers  which,  in  turn,  run  on  all  major  platforms 
including  Windows  PCs,  Macintoshes,  and  UNIX 
systems. 

Server  applications  and  control  should  be  built  on  a 
Windows  NT  platform  running  Microsoft’s 

Exchange  Server  and  SQL  Server. 

•  Support  for  major  protocols  and 
ease  of  integration  with  other 
applications  that  support  those 
protocols. 

•  %  market-share  of  the  platforms 
on  which  the  applications  and 
development  environment  will 
run. 

4 

Flexible 

The  value  scorecard  and  design  worksheets  should 
be  tailorable  to  specific  program  needs. 

Scorecard  columns  can  be  added,  removed,  or 
further  subdivided.  Calculations  between  the 
columns,  including  dependency  adjustments,  should 
be  fully  user-configurable. 

Worksheet  organization  and  content  should  be  fully 
user  configurable. 

Links  between  the  scorecard  and  the  underlying 
design  worksheets  should  be  configurable. 

Scorecard  and  worksheets  should  easily  provide  data 
to,  and  accept  data  from,  other  applications. 

•  User  feedback  &  response 
ratings  on  how  easily  and 
effectively  the  scorecards  and 
worksheets  can  be  configured 
for  specific  S&T  programs. 

•  User  feedback  &  response 
ratings  on  ease  of  reconfiguring 
scorecards  and  worksheets  after 
the  initial  construction  and 
partial  population. 
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Table  11.  (Concluded) 


.  #  ■ 

Requirement 

Description/Features 

How  Measured 

5 

Security 

Sensitive  data  should  be  kept  secure  over  the 
network  via  auto  and  point-to-point  private 
encryption.  Support  for  classified  data  is  not 
required. 

Password  protection  for  sensitive  areas  and 
applications. 

•  Granularity  of  password  and 
encryption  protection  capability. 

•  Robustness  of  password  (e.g. 
encrypted  passwords)  and  other 
encryption  algorithms. 

6 

Technology  Transition 
Cost  and  Life  Cycle 
Estimation  during 

S&T 

During  technology  development,  the  tools  should  aid 
in  estimating  the  relative  and  delta  costs  to  transition 
the  technology  to  a  weapon  system  acquisition 
activity. 

The  tools  should  aid  in  estimating  the  delta  life  cycle 
cost  impacts  associated  with  a  new  technology. 

•  Near  Term:  Model  granularity 
and  fidelity  of  “as-like” 
estimates. 

•  Long  Term;  Agreement 
between  projected  and  measured 
values. 

3.4.2.3  Recommendations  for  Future  Development 

As  indicated  earlier,  value  analysis  is  multidisciplinary  and,  therefore,  involves  the  integration  of 
a  number  of  different  applications.  Value  analysis  does  not  require  a  “value  analysis  tool”  but 
rather  a  “toolkit”  comprised  of  several  core  applications  and  the  “middleware”  required  to 
interface  them  to  a  variety  of  external  applications.  The  toolkit  itself  must  be  flexible,  so  that 
new  or  different  applications  could  be  easily  incorporated  into  the  suite  as  required.  The 
middleware  must  support  the  integration  of  tools  that  capture  information  from  the  design 
process  with  tools  that  can  support  its  representation  in  technology  worksheets.  It  must  also 
support  value  scorecard  organization  and  tailoring,  as  well  as  the  interface  between  the 
worksheets  and  the  scorecard.  That  interface  is  bi-directional.  Rolled  up  values  from  the 
worksheets  are  recorded  in  the  scorecard,  and  sensitivity  analyses  performed  at  the  scorecard 
level  must  access  the  underlying  worksheet  information. 

The  value  analysis  activity  in  the  S&T  IPPD  process  is  central  to  establishing  a  quantitative  basis 
for  assessing  the  relative  value  of  alternative  technologies.  The  bottom  line  resulting  from  the 
scorecard  should  be  an  assessment  of  the  cost  and  risk  associated  with  technology  transition  from 
the  laboratory  to  the  acquisition  community  or  to  the  defense  industrial  base.  While  the 
scorecard  provides  an  indication  of  technology  maturity  on  the  basis  of  that  transition  cost  and 
risk,  there  exists  today  no  commonly  agreed-on  set  of  metrics  or  Key  Performance  Indicators 
(KPIs)  for  technology  maturity. 


Three  projects  are  recommended  to  support  the  need  for  improved  tools  for  value  analysis.  They 
include  the  need  for  a  toolkit,  the  need  for  a  set  of  agreed-upon  KPIs  for  technology  maturity, 
and  the  need  to  extend  the  heuristic  process  capability  models  developed  in  the  electronics 
domain  to  other  domains  (e.g.,  mechanical,  composites). 
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3.4.3.1  Recommendation  1:  Value  Analysis  Toolkit 

Background:  The  two  key  needs  of  a  value  analysis  toolkit  center  on  application  integration  and 
worksheet/scorecard  support  functions.  Application  integration  is  required  to  bring  data  together 
from  disparate  analyses.  In  the  area  of  performance,  the  data  might  result  from  calculations, 
simulations,  mathematical  models,  heuristic  models,  expert  consensus,  or  other  sources.  The 
same  is  true  in  the  areas  of  producibility,  reliability,  supportability,  etc.  Parametric  cost  models 
or  other  cost  modeling  techniques  might  be  used  to  develop  transition,  scale-up,  support,  or  other 
life-cycle-cost  estimates.  (In  the  case  of  a  new  technology,  the  analyses  would  focus  on  the 
potential  impact  of  the  technology  on  those  cost  factors  should  it  be  implemented,  generally 
yielding  a  relative  or  delta  cost.)  The  information  from  these  analyses  is  captured  in  various 
design/technology  worksheets.  Aside  from  simple  spreadsheets,  the  “worksheet  application” 
does  not  exist.  A  really  helpful  worksheet  application  would  be  a  “worksheet  assistant”  that 
would  provide  interactive  guidance  to  the  user  in  terms  of  worksheet  tailoring,  understanding  the 
most  important  factors,  capturing  the  rationale/sources  behind  the  data,  and  supporting  the  roll¬ 
up  to  the  value  scorecard  rating.  The  roll-up  function  would  “tag”  its  own  “audit  trail”  so  that  a 
user  could  quickly  see  and  imderstand  the  contributions  to  the  rolled-up  number.  It  would  also 
generate  the  “hooks”  to  enable  subsequent  “backtracking”  and  “what-if  ’  scenario  development 
associated  with  sensitivity  analysis. 

The  other  component  that  is  needed  in  value  analysis  is  a  “scorecard  assistant”  that  is  analogous 
to  the  worksheet  assistant  just  described.  The  scorecard  application  would  assist  users  in 
tailoring  the  scorecard  to  their  needs.  It  would  provide  access  to  instructions  and  examples  on 
filling  in  the  scorecard,  instructions  and  examples  on  sensitivity  analysis,  statistical  and  graphical 
support  for  analyzing  scorecard  results,  and  guidance  on  using  the  scorecard  as  a 
management/investment  decision  tool.  The  scorecard  assistant  must  be  able  to  “talk  to”  the 
worksheet  assistant.  The  two  must  work  interactively  to  support  the  user  in  determining  the 
significance  of  and  uncertainties  associated  with  various  scorecard  values. 

Purpose:  The  purpose  of  this  effort  is  to  develop  a  prototype  value  analysis  toolkit  that  will 
support  the  AFMC  S&T  IPPD  value  analysis  activities.  This  effort  would  result  in  a  prototype 
toolkit  that  is  validated  against  actual  S&T  projects.  The  toolkit  would  be  used  to  demonstrate  a 
quantitative  assessment  of  the  relative  value  of  competing  technologies  using  S&T  value  analysis 
methods. 


Level  of  Effort  and  Timing:  The  recommended  timing  and  level  of  effort  (in  person-years)  for 
this  program  are  as  follows. 


FY97  FY98  FY99 

Person-Hours  5000  5000  7000 


FYOO 

7000 


Deliverables 


Detailed  Design 
&  Concept 
Demonstration 


Worksheet  & 
Scorecard 
Assistant  Design 


“  Assistant" 
Development  & 
Demonstration 


“Assistant” 
Evolution  & 
Applications 
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3.4.3.2  Recommendation  2:  Key  Performance  Indicators  (KPIs)  for  Technology  Maturity 

Background:  One  of  the  fundamental  objectives  of  the  AFMC  S&T  IPPD  Initiative  is  to  ensure 
that  the  technology  which  emerges  from  the  end  of  the  technology  “pipeline”  is  sufficiently 
mature  to  be  cost-effectively  transitioned  into  the  target  user  community.  The  problem  is  that 
there  is  no  common  understanding  of  the  notion  of  “technology  maturity”  either  in  S&T  or  in  the 
acquisition  community. 

Purpose:  The  objective  of  this  effort  is  to  work  \vith  both  academia  and  industry  to  develop  a  set 
of  KPIs  that  could  be  used  to  reliably  evaluate  the  “readiness  for  transition”  of  a  wide  range  of 
technologies  emerging  from  S&T.  Ultimately,  these  indicators  should  provide  potential 
technology  implementers  with  a  credible  assessment  of  the  risks  associated  with  using  a  new 
technology  in  a  product  design  and  development  effort.  This  effort  should  address  software  as 
well  as  hardware  technologies. 

Level  of  Effort  and  Timing:  The  recommended  timing  and  level  of  effort  (in  person-years)  for 
this  program  are  as  follows. 


FY97 

FY98 

FY99 

FYOO 

Person-Hours 

400 

3500 

4000 

Deliverables 

Initial  Problem 
Description  & 
Approach 

Initial  KPIs  for 
New 

Technologies 

Refined  Set  of 
Technology  KPIs 
and  Application 
Examples 

3.4.53  Recommendation  3:  Heuristics-Based  Design-Process  Simulation  Models 

Background:  Leading  companies  in  the  electronics  domain  have  initiated  the  development  of 
heuristic  models  that  implement  new  designs  in  a  “virtual  build”  in  order  to  improve  design 
robustness  and  producibility  prior  to  their  manufacture.  These  models  are  more  than  “design 
rules.”  They  relate  process  capability  and  expected  defects  to  design  features,  and  provide  real¬ 
time  feedback  to  designers  regarding  the  implications  of  various  design  decisions.  These  models 
are  only  now  being  fleshed  out  in  the  electronics  domain  for  specific  families  of  products,  but 
early  indicators  suggest  that  the  potential  cost  and  cycle-time  savings  from  employing  such 
models  in  the  design  process  are  enormous. 

Purpose:  Understand  the  heuristic  modeling  approach  as  it  is  evolving  in  the  electronics  domain, 
and  explore  the  feasibility  of  applying  the  same  principles  to  the  airframe  and  propulsion 
domains. 

Level  of  Effort  and  Timing:  The  recommended  timing  and  level  of  effort  (in  person-years)  for 
this  program  are  as  follows. 
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FY97 

FY98 

FY99 

FYOO 

FY01 

Person-Hours 

300 

6500 

8000 

8000 

5000 

Deliverables 

Framework  for 

Development  of 

Development  of 

Domain- 

heuristic  models 

models  and 

models  and 

Independent 

from  electronics 

demo  for 

demo  for 

Framework  for 

airframe 

propulsion 

Heuristic  Models 
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4.  Tools  Deployment  Strategy 


The  end-user  assessment  of  the  S&T  IPPD  tool  areas  revealed  that  program  managers  need  tools 
to  implement  the  S&T  IPPD  process.  This  assessment  also  showed  that  each  program  is  likely  to 
have  different  needs  depending  on  the  natiue  of  the  research  and  the  maturity  of  the  technology 
under  development.  Needs  identified  by  the  participants  in  the  interviews  were  used  to  conduct 
market  assessments  for  each  tool  area.  Recommended  deployment  approaches  were  given  for 
each  tool  area  in  section  3.  In  addition  to  the  deployment  strategies  delineated  in  section  3, 
certain  activities  need  to  continue  throughout  the  S&T  IPPD  tools  and  methods  effort.  These 
activities  are: 


•  Minimize  the  commitment  to  customize  or  combine  tools  until  the  need  is  imminent  in  an 
S&T  program. 

•  Monitor  market  developments  regarding  the  key  features  and  capabilities  identified  by 
the  users  during  the  interviews. 

•  Seed  protot5T)ing  software  development  in  the  vailue  analysis  area  since  the  marketplace 
is  only  now  recognizing  this  tool  area  as  a  potential  product  category. 

•  Encourage  and  participate  in  standards  development,  particularly  in  the  areas  of  Web- 
enabled  workflow,  requirements  analysis,  and  security. 

•  Form  a  Tools  and  Methods  Working  Group  to  track,  assess,  and  report  on  tool  and 
method  developments  related  to  the  S&T  IPPD  process. 

It  is  stressed  again  that  this  deployment  plan  is  not  a  software  development  plan.  The  intent  is  to 
use  COTS  tools  to  meet  as  many  user  needs  as  possible  before  modification  is  considered. 
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5.  Summary 


The  primary  objective  of  this  effort  was  to  gather  user  needs  for  tools  and  methods  to  support  the 
implementation  of  the  S&T  IPPD  process.  This  report  has  presented  the  results  of  the  research 
conducted  for  four  tool  areas:  (1)  (Technology)  Requirements  Collection,  Organization,  and 
Analysis;  (2)  Value  Judgment  via  Expert  Consensus;  (3)  (Program  Management)  Workflow;  and 
4)  (Design)  Value  Analysis.  The  research  approach  was  to  first  interview  users  to  collect  and 
verify  user  needs  for  the  tools.  Next,  a  market  assessment  was  accomplished  to  determine  if 
COTS  tools  exist  that  fulfill  the  users’  needs.  Finally,  a  strategy  was  developed  for  tools 
deployment  throughout  the  Air  Force  S&T  community. 

The  user  interviews  were  conducted  using  Ventana  GroupSystems  software.  The  interviews 
were  structured  as  information-sharing  sessions  where  the  research  team  presented  information 
about  which  the  users  were  asked  to  provide  feedback  and  comments.  This  methodology  proved 
to  be  effective  in  capturing  and  validating  needs  for  a  Group  Consensus  Tool. 

Once  the  user  needs  were  known,  a  market  assessment  was  conducted.  The  market  assessment 
showed  that  there  is  an  abtmdant  supply  of  tools  for  all  areas  except  value  analysis.  This  is  good 
news  for  users  since  competition  will  produce  more  and  better  tools.  Since  different  users  ■will 
probably  have  different  tool  needs,  they  can  each  use  the  evaluation  matrices  shown  for  each  tool 
area  to  choose  a  tool  based  on  their  particular  needs. 

This  report  recommends  a  tools  development  strategy  for  deploying  tools  throughout  the  S&T 
community.  As  such,  this  report  is  not  a  software  development  plan,  but  a  strategy  for  selecting 
tools,  customizing  them,  and  integrating  them  over  time  into  specific  S&T  programs  having 
specific  objectives.  The  essence  of  the  strategy  is: 

•  Choose  selected  pilot  programs  to  implement  the  S&T  IPPD  process. 

•  Minimize  the  commitment  to  customize  or  combine  tools  until  the  need  is  imminent  in  an 
S&T  program. 

•  Monitor  market  development  regarding  the  key  features  and  capabilities  identified  by 
users  in  the  interviews. 

•  Seed  prototyping  software  development  in  value  analysis  since  the  marketplace  is  only 
now  recognizing  this  tool  area  as  a  potential  product  category. 

•  Encourage  and  participate  in  standards  development,  particularly  in  the  areas  of  Web- 
enabled  workflow,  requirements  analysis,  and  security. 

•  Form  a  Tools  and  Methods  Working  Group  to  track,  assess,  and  report  on  tools  and 
methods  developments  related  to  the  S&T  IPPD  process. 

Follow-on  work  is  necessary  to  choose,  modify,  and  field  tools  for  6.3  program  managers  to  use 
in  their  everyday  activities.  This  effort  should  be  closely  coordinated  ■with  the  overall  IPPD 
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initiative  to  maximize  benefits  for  users.  The  strategy  delineated  in  this  report  represents  the  first 
step  in  deploying  tools  and  methods  to  assist  program  managers  in  implementing  the  S&T  IPPD 
process.  By  employing  IPPD  principles  and  practices,  the  S&T  culture  can  move  away  from  the 
historic  performance-at-any-cost  approach  to  technology  development  and  application,  toward  a 
new,  more  cost-effective  and  risk-managed  approach. 
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